Visual question and answer model training method and system based on dynamic loss depolarization

Abstract

The invention discloses a dynamic loss depolarization-based visual question and answer model training method and system, which comprises the following steps: dynamically adjusting the margin parameter of a loss function according to the answer frequency of training data and a sample deviation detection result, generating a dynamic margin inversely proportional to the answer frequency, and generating a dynamic margin inversely proportional to the deviation degree by introducing an independent deviation detection branch and predicting the deviation confidence degree based on the question text in sample deviation detection; performing weighted fusion on the dynamic margin based on the answer frequency and the dynamic margin based on deviation detection to form a final dynamic margin; and using the fused dynamic margin loss function to optimize a visual question and answer model, and performing joint training in combination with cross entropy loss and loss of a deviation detection branch. According to the invention, the prediction capability of the model on long-tail distribution answers can be enhanced, the recognition accuracy of low-frequency answers is improved, and the overall generalization performance of the model on visual question and answer tasks is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of visual question answering model training based on dynamic loss debiasing, and in particular to a visual question answering model training method and system based on dynamic loss debiasing. Background Art

[0002] Visual Question Answering (VQA) is a multimodal AI task that enables computers to generate correct answers based on input images and questions. It is widely used in fields such as autonomous driving, intelligent customer service, and medical image analysis. A VQA model typically consists of an image feature extractor and a text feature encoder, which then fuses these features to predict the answer.

[0003] In practical applications, VQA models are often affected by dataset bias. The bias mainly comes from:

[0004] Unbalanced answer distribution: Some answers appear much more frequently than others in the dataset, causing the model to tend to predict high-frequency answers and ignore the actual content of the image and question.

[0005] Question language bias: Some questions have obvious language cues, which enable the model to make more accurate predictions without the need for image information.

[0006] Insufficient visual information: When the target object in the image is small or there is too much background information in a complex scene, it is difficult for the model to effectively extract key information.

[0007] In order to solve the problem of dataset bias, researchers have proposed a variety of methods to address the shortcomings of existing solutions:

[0008] Based on data enhancement: Balance the data distribution by transforming the image or question, but this cannot completely solve the problem of high-frequency answer tendency.

[0009] Based on reweighting or resampling: giving higher weights to low-frequency answers or oversampling, which may lead to degraded model performance and unstable training.

[0010] Adversarial training: Adaptive networks are used to generate unpredictable samples to reduce bias, but training is complex and requires additional computing resources.

[0011] When dealing with data set biases (such as category imbalance and long-tail distribution), traditional models rely heavily on preset fixed parameters (such as fixed margins), which limits the model's generalization ability and severely restricts its robustness. Summary of the Invention

[0012] In order to solve the above technical problems in the prior art, the present invention proposes a visual question answering model training method and system based on dynamic loss debiasing to solve the above technical problems.

[0013] According to a first aspect of the present invention, a method for training a visual question answering model based on dynamic loss debiasing is proposed, comprising:

[0014] S1: Dynamically adjust the margin parameter of the loss function based on the answer frequency of the training data and the sample bias detection results. The answer frequency is calculated by counting the number of times each answer appears under a given question type, generating a dynamic margin inversely proportional to the answer frequency. The sample bias detection introduces an independent bias detection branch, predicts the bias confidence based on the question text, and generates a dynamic margin proportional to the degree of bias.

[0015] S2: Weighted fusion of the dynamic margin based on answer frequency and the dynamic margin based on deviation detection to form the final dynamic margin;

[0016] S3: Use the fused dynamic margin loss function to optimize the visual question answering model, and combine the cross entropy loss and the loss of the bias detection branch for joint training.

[0017] In some specific embodiments, the dynamic margin calculation based on the answer frequency is specifically as follows: in, Represents the answer a for a given question type in the training set i Frequency of occurrence, m freq [i] represents the dynamic margin of the answer ai based on the given question type, ε is a hyperparameter to avoid computational overflow, and |Ω| represents the full set of the answer space.

[0018] In some specific embodiments, under a given question type, for each answer a i , using a random version of the dynamic margin, and finally a dynamic margin based on the answer frequency Where N is the Gaussian distribution and σ is the standard deviation.

[0019] In some specific embodiments, the deviation detection branch is a unimodal branch that uses only the question text, and the output is converted into deviation confidence through a Softmax function.

[0020] In some specific embodiments, the formula for the dynamic margin of sample deviation detection is m q [i]=softmax(f q / τ), τ represents the temperature coefficient, f q is the confidence of the deviation detection branch prediction, f q =e q (fbias ), e q is the classification head for prediction, f bias is the predicted answer distribution based on the dataset bias captured by the bias branch according to the question information, fbias = f(QW), where Q represents the question text and W is a learnable mapping parameter.

[0021] In some specific embodiments, in S2, the dynamic margin based on the answer frequency and the dynamic margin based on the deviation detection are weighted and fused to form the final dynamic margin m comb [gt] = βm ran [gt]+(1—β)m q [gt], gt represents the index of the true label of the sample, β is a hyperparameter used to measure the two margins, β is set to 1 at the beginning of training and gradually decays linearly with the training rounds.

[0022] In some specific embodiments, the total loss function of the visual question answering model is: L total =L fre +L vqa +L q +L comb , where L fre is the loss function based on answer frequency, L comb is the adjusted cosine loss, L vqa is the cross entropy loss of the main branch of visual question answering, L q is the loss of the deviation detection branch.

[0023] In some specific embodiments, Among them, a i is the weight factor of category i, scos() is the cosine similarity function, θ i represents the angle representation of the feature vector of category i, N represents the total number of training samples, t i represents the true label of the i-th sample, σ() is the sigmoid function, f(v i ,q i ) is the neural network model, v i is the image feature input, q i is the question text feature input.

[0024] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, the above-mentioned method is implemented.

[0025] According to a third aspect of the present invention, a visual question answering model training system based on dynamic loss debiasing is proposed, comprising:

[0026] Dynamic Margin Loss Calculation Unit: This unit is configured to dynamically adjust the margin parameters of the loss function based on the answer frequency of the training data and the sample bias detection results. The answer frequency is calculated by counting the number of times each answer appears under a given question type, generating a dynamic margin that is inversely proportional to the answer frequency. The sample bias detection introduces an independent bias detection branch, predicts the bias confidence based on the question text, and generates a dynamic margin that is proportional to the degree of bias.

[0027] Loss function fusion unit: configured to weightedly fuse the dynamic margin based on answer frequency and the dynamic margin based on deviation detection to form a final dynamic margin;

[0028] Model training optimization unit: Configured to optimize the visual question answering model using the fused dynamic margin loss function and perform joint training with the cross entropy loss and the loss of the bias detection branch.

[0029] The present invention proposes a visual question answering model training method and system based on dynamic loss debiasing. By introducing a dynamic loss debiasing mechanism, the bias problem in the data set is effectively alleviated during the visual question answering model training process. A greater penalty is imposed on high-frequency answers by using a margin loss based on the frequency of the answer, and at the same time, the dynamic margin calculated by the deviation detection branch is used to provide more learning space for low-frequency answers. This application not only improves the robustness of the model under long-tail distribution, but also significantly improves the recognition accuracy of low-frequency answers. In addition, by jointly optimizing the backbone visual question answering loss and the deviation detection loss, the model reduces the dependence on language deviation while maintaining an efficient fusion of image and text information, thereby achieving better performance in a variety of visual question answering tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many of the intended advantages of the embodiments will be readily apparent as they become better understood by reference to the following detailed description. Other features, objects, and advantages of the present application will become more apparent upon reading the detailed description of the non-limiting embodiments made with reference to the following drawings:

[0031] Figure 1 This is a flowchart of a visual question answering model training method based on dynamic loss debiasing according to an embodiment of the present application;

[0032] Figure 2 This is a framework diagram of a visual question answering model training system based on dynamic loss debiasing according to a specific embodiment of the present application;

[0033] Figure 3This is a comparison diagram of the SoftMax function and the decision boundary of the regularized function in a specific embodiment of the present application;

[0034] Figure 4 This is a comparison diagram of the decision boundaries of the loss function based on a fixed margin and a dynamic margin in a specific embodiment of the present application;

[0035] Figure 5 This is a performance comparison chart of the scaling factor scale of a specific embodiment of the present application on VQA-CP v1 and VQA-CP v2;

[0036] Figure 6 This is a t-SNE visualization of the answer feature space of a specific embodiment of the present application on the VQA-CP v1 dataset;

[0037] Figure 7 This is a t-SNE visualization of the answer feature space of a specific embodiment of the present application on the VQA-CP v2 dataset;

[0038] Figure 8 This is a graph showing the accuracy of a model of a specific embodiment of the present application on VQA-CP v2 for five specific question types;

[0039] Figure 9 This is a visual representation of a typical example of a specific embodiment of the present application;

[0040] Figure 10 This is a comparison diagram of answer feature distribution of different models in cosine space in a specific embodiment of the present application;

[0041] Figure 11 This is a framework diagram of a visual question answering model training system based on dynamic loss debiasing according to an embodiment of the present application;

[0042] Figure 12 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0043] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0044] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] Figure 1 FIG1 shows a flow chart of a method for training a visual question answering model based on dynamic loss debiasing according to an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0046] S101: Dynamically adjust the margin parameters of the loss function according to the answer frequency of the training data and the sample deviation detection results. The answer frequency generates a dynamic margin inversely proportional to the answer frequency by counting the number of occurrences of each answer under a given question type. The sample deviation detection generates a dynamic margin proportional to the degree of deviation by introducing an independent deviation detection branch and predicting the deviation confidence based on the question text.

[0047] In a specific embodiment, it is known that bias in visual question answering tasks arises when similar questions in a dataset are often paired with frequently asked answers, causing the model to completely ignore the images and instead memorize the association between the question and the answer. As a result, some samples corresponding to infrequent answers are ignored, while other samples receive excessive attention. Dynamic margin allows the model to set different penalties for different samples, thereby providing highly discriminative features for different training samples. It ensures that samples with frequently asked answers are allowed a smaller margin than samples with infrequent answers.

[0048] In a specific embodiment, the dynamic margin calculation based on the answer frequency is specifically as follows: in, Represents the answer a for a given question type in the training set i Frequency of occurrence, m freq [i] represents the dynamic margin of the answer ai based on the given question type, ∈ is a hyperparameter to avoid computational overflow, and |Ω| represents the full set of answer spaces. Loss function For a given problem type qt k , high-frequency answers have wider spans (smaller margins) in cosine space, while low-frequency answers have narrower spans (larger margins) in cosine space. Specifically, high-frequency answers mean more training samples, which requires a wider feature space to fully cover. On the contrary, for low-frequency answers, a tighter feature space helps model inference. For a given question type qt in the training set k The dynamic margin m based on the answer frequency freq [i] For each a i Is fixed. When the problem type is qt k When it is "howmany", for the problem type qt kFor all training samples of , the margin for the answer "2" is the same. In addition, setting a low margin for high-frequency answers under all given question types and a high margin for low-frequency answers under all given question types is likely to overcorrect the bias, forcing the model to pay more attention to low-frequency answers rather than some common answers, which ignores the common sense learning in visual question answering tasks. Therefore, for a given question type qt k Next, for each answer a i , all use m freq A random version m of [i] ran [i], ultimately dynamic margin based on answer frequency Where N is the Gaussian distribution and σ is the standard deviation, which is a hyperparameter. This prevents the model from overcorrecting for low-frequency answers.

[0049] In a specific embodiment, simply randomizing the average answer frequency cannot well take into account the degree of deviation of each sample. Therefore, a deviation detection branch is introduced to capture the degree of deviation of the sample: f bias =f(QW),f q =e q (f bias ), so that the model can more accurately distinguish samples with different degrees of deviation and thus better adjust the decision boundary. bias is the predicted answer distribution of the dataset bias captured by the bias branch according to the question information, and f q is the confidence of the deviation detection branch prediction (i.e., prediction probability), e q is the classification head for prediction, Q represents the question text, and W is the learnable mapping parameter. The formula for the dynamic margin of sample deviation detection is m q [i]=softmax(f q / τ), τ represents the temperature coefficient, which is a hyperparameter of the network. q [i] is the learnable margin for bias, obtained during model training. When the sample is unbiased, this margin is similar to the margin based on answer frequency. However, for biased samples, the greater the degree of bias, the larger the dynamic margin, leaving more room for the model's predictions.

[0050] S102: Weighted fusion of the dynamic margin based on the answer frequency and the dynamic margin based on the deviation detection to form a final dynamic margin.

[0051] In a specific embodiment, the final dynamic margin m comb [gt] = βm ran [gt]+(1—β)m q[gt], gt represents the index of the true label of the sample, β is a hyperparameter used to measure the two margins, β is set to 1 at the beginning of training and gradually decays linearly with the training rounds. The basic question answering model uses the adjusted cosine loss function Parameter optimization is performed, where a i is the weight factor of category i, scos() is the cosine similarity function, θ i Angular representation of the feature vector representing category i.

[0052] S103: Use the fused dynamic margin loss function to optimize the visual question answering model, and combine the cross entropy loss and the loss of the bias detection branch for joint training.

[0053] In a specific embodiment, Figure 2 This is a framework diagram of a visual question answering model training system based on dynamic loss debiasing in a specific embodiment of the present application. Figure 2 As shown in the figure, a bias detection branch (BiasDetectionBranch) is introduced in the model architecture layer. In the bias detection branch, a "QuestionOnly" branch is introduced. This branch only uses a single modality (text) data for training, so it can better capture the deviation in the data set. Samples that are usually easier to get the correct answer through text data alone are often simple samples or samples with larger deviations. The larger the predicted probability value, the higher the degree to which the branch uses the deviation of the data set for prediction. This branch and the basic model both use the binary cross entropy loss function L vqa and L q Parameter optimization is performed, where N represents the total number of training samples, t i represents the true label of the i-th sample, σ() is the sigmoid function, f(v i ,q i ) is the neural network model, v i is the image feature input, q i It is the question text feature input. It is worth noting that the “Question Only” branch does not back propagate to the text encoder e when performing gradient back propagation. q , avoiding the text encoder e q Learning the dataset bias will not affect the gradient backpropagation and parameter update of the backbone model. The total loss function of the visual question answering model is: L total =Lf re +L vqa +L q +L comb .

[0054] Figure 3 This is a comparison diagram of the SoftMax function and the decision boundary of the regularized function in a specific embodiment of the present application. Figure 4 This is a comparison diagram of the decision boundaries of the loss function based on a fixed margin and a dynamic margin in a specific embodiment of the present application, such as Figure 3 and Figure 4 As shown, the green area represents the minority class, while the orange area represents the frequent class (i.e., the dominant class). The traditional SoftMax function has the limitation of treating all samples equally when calculating the loss value. Especially in datasets with severe class imbalance, the SoftMax function can cause the overall loss value to be dominated by the answer class with a higher proportion, thus affecting the effectiveness of model training. Regularization allows the model to pay more attention to the minority class during training. By normalizing the weight matrix and eigenvectors, the model pays more attention to pairs of samples with large angular dissimilarity in feature space. In cosine space, similar samples correspond to cosine values closer to 1, while dissimilar samples correspond to cosine values closer to 0. Therefore, samples in the minority class often have large angular dissimilarity from samples in the frequent class (i.e., the dominant class) in feature space, resulting in cosine values closer to 0. Through normalization, the model places greater emphasis on these smaller cosine values, thereby giving more weight to samples in the minority class. This improves the model's ability to learn from the minority class and alleviates the class imbalance problem. When a fixed cosine margin is used in the presence of dataset bias, the answer distributions for each question type exhibit significant deviations, which prevents the model from learning sufficient feature representations in the cosine space. Specifically, the fixed cosine margin approach suffers from the following issues when dealing with dataset bias: Since the answer distributions for different question types exhibit significant deviations, the fixed cosine margin fails to fully account for these differences. Consequently, the model may not accurately distinguish the boundaries between different question types in the cosine space, resulting in degraded classification performance. The method proposed in this application introduces a dynamic cosine margin, dynamically adjusting the cosine margin to reflect the characteristics of the answer distributions for different question types. This approach allows the model to more flexibly adapt to the differences between different question types and is more robust to dataset bias. The model trained in this application searches for answers in feature spaces of different sizes for different question types. For minority categories in the green area, the model searches for answers in a smaller feature space to more accurately distinguish these minority categories. For frequent categories in the orange area, the model searches for answers in a larger feature space to ensure correct classification of the majority category.

[0055] In order to verify the performance of the visual question answering model based on dynamic loss debiasing in this application, the VQA-CPv1 and VQA-CPv2 datasets were selected for experiments. The VQA-CP dataset is an out-of-distribution benchmark dataset used to evaluate the performance of the model on unseen datasets or distributions. The experiment was deployed on the Linux (Ubuntu18.04) operating system, and the model was built using the PyTorch (1.0.1) deep learning framework, and the model was trained on an RTX2080Ti (11GB). For the question length, the maximum length is set to 14. For the image dataset, MSCOCO, which is a widely used image dataset, was selected. In terms of preprocessing images, the number of objects extracted from each image was 36, and 2048 features were extracted from each extracted object area. For the model parameters, the default value of entropy is set to 4.5, the default value of the scaling factor is set to 16, and the default value of the temperature parameter is set to 0.2. Among them, the entropy value is a decision criterion to decide whether to reset the feature question type qt k The answer counts under ,or retain them, depends on the diversity of the answer distribution.,In addition, the number of samples (images, questions, and answers) used on the VQA-CPv1 and VQA-CPv2 datasets are shown in,Table 1.

[0056] Table 1. Sample numbers of VQA-CP v1 and VQA-CP v2

[0057]

[0058] The experimental results are shown in Tables 2 and 3. The model accuracy is divided into the overall accuracy (All), the accuracy of the "yes or no" question type (Y / N), the accuracy of the "count" question type (Num), and the accuracy of the "other" question type (Other).

[0059] Table 2 Experimental results of the visual question answering model based on dynamic loss debiasing on VQA-CP v1

[0060]

[0061]

[0062] The model based on dynamic loss debiasing proposed in this application uses UpDn as the base model for visual question answering tasks. The experimental results on the VQA-CPv1 dataset are shown in Table 2. In terms of the overall accuracy of "Overall", the model achieved excellent performance results and obtained an absolute improvement of 25.39% compared with the basic model UpDn (63.41% vs 38.02%). More importantly, the model of this application exceeded the Question-only, SAN, NMN, MCB, Strong-BL, Counter, GVQA, AdvReg, LMH and AdaVQA models by 43.25%, 36.53%, 33.77%, 29.02%, 29%, 25.74%, 24.18%, 19.98%, 8.65% and 7.49% respectively. More notably, the model significantly surpassed the previous baseline model in terms of the "Yes / No" question type, achieving an accuracy of up to 90.49%. The “Yes / No” question type usually has a clearer and more direct semantic logic. The dynamic loss designed by this application can dynamically adjust the weights so that a minority of categories receive more attention during training, effectively compensating for the dataset bias and improving the model’s generalization ability for all categories. In addition, the model of this application achieves suboptimal and optimal performance in the “Number” and “Other” question types, respectively, further demonstrating the model’s outstanding performance in dealing with diverse problems. For the “Number” question type, the model has the ability to effectively understand and process numerical reasoning, thereby achieving excellent performance. For the “Other” question type, the model uses a dual-model structure and flexible exploration of the answer space to comprehensively understand the context and semantic information. Therefore, the results of the model on the VQA-CPv1 dataset show that the model has strong application potential in dealing with various complex tasks and scenarios.

[0063] In order to further verify the effectiveness of the debiasing of the model of this application, experiments were conducted on another out-of-distribution benchmark dataset VQA-CPv2, and the results are shown in Table 3. The benchmark models are divided into two categories, one is the basic model commonly used in visual question answering tasks, and the other is the method based on the UpDn model. The model of this application achieved the same excellent performance on the VQA-CPv2 dataset, and exceeded the Question-only, SAN, NMN, MCB, Strong-BL, Counter, UpDn, GVQA, AdvReg, Rubi, LMH, VGQE, CSS, Decomp-LR and AdaVQA models by 41.6%, 32.59%, 30.08%, 21.22%, 23.14%, 19.88%, and 10.9% respectively in terms of overall accuracy.

[0064] 16.76%, 26.25%, 16.38%, 10.44%, 5.4%, 7.44%, 16.39%, 8.68% and 3.55%. On the "Yes / No" question type, a high accuracy of 88.86% was also achieved, which once again verified the effectiveness and robustness of dynamic loss in dealing with unbalanced data sets. By dynamically adjusting the loss weights, the model of this application is able to focus on the two categories of "Yes" and "No" during training, and ensure that the model fully considers the importance of minority categories during the learning process. In addition, the model achieved suboptimal performance on the "Number" question type, but average performance on the "Other" question type. This may be due to the characteristics of the VQA-CPv2 dataset. In the "Other" question type, the semantics and context involved are more diverse and complex, and include a wider range of topics, which increases the difficulty of model processing.

[0065] Table 3 Experimental results of the visual question answering model based on dynamic loss debiasing on VQA-CP v2

[0066]

[0067] In a specific embodiment, detailed ablation experiments were conducted on both the VQA-CPv1 and VQA-CPv2 datasets. Figure 5 A performance comparison diagram of the scaling factor of a specific embodiment of the present application on VQA-CPv1 and VQA-CPv2 is shown, as shown in FIG. Figure 5 As shown in , the horizontal axis represents the scale, and the vertical axis represents the overall accuracy. The blue line represents the experimental data on the VQA-CPv1 dataset, and the red line represents the experimental data on the VQA-CPv2 dataset. Figure 5 It can be seen that the same model achieves optimal performance at different scaling scales. When the scaling scale is 6, the model achieves an accuracy of 63.41% on the VQA-CPv1 dataset; when the scaling scale is 8, the model achieves an accuracy of 57.55% on the VQA-CPv2 dataset. It is worth noting that when the scaling scale is set to 0, the model performance is almost equivalent to that of the base model UpDn. When the scaling scale is set to 32, the model achieves an accuracy of 59.56% and 55.43% on the VQA-CPv1 and VQA-CPv2 datasets, respectively, which are 3.85 and 2.21 percentage points lower than the optimal performance, respectively. This shows that a scale that is too small is insufficient to learn the feature space, while a scale that is too large will also lead to unsatisfactory results.

[0068] In a specific embodiment, component ablation experiments were conducted on VQA-CPv1 and VQA-CPv2 to evaluate the contribution of each model component. The experimental results were evaluated based on overall accuracy, accuracy for yes / no questions, accuracy for count questions, and accuracy for other question types. The ablation results of each model component on the VQA-CPv1 dataset are shown in Table 4. The model of this application is built on the UpDn model structure, adding a "question-only" branch to UpDn and using a dynamic loss adjustment method. Based on the aforementioned experimental results, the scaling factor s was set to 6. The second row of Table 4 shows that adding a bias detection branch to UpDn increases the overall accuracy by 4.06%. The bias detection branch helps the model identify biases in the dataset during training and provides additional information to adjust the loss weights. The third row of Table 4 shows that replacing the binary cross entropy loss function with a cosine dynamic loss based on answer frequency increases the overall accuracy by 25.32% compared to the base model UpDn. This method dynamically adjusts the loss weights based on the frequency of the answer category, allowing the model to pay more attention to the minority category. The fourth row of Table 4 shows that replacing the binary cross-entropy loss function with a cosine dynamic loss based on answer frequency and bias detection yields a 25.39% improvement in overall accuracy compared to the base model UpDn. The cosine loss, combined with answer frequency and bias detection, comprehensively accounts for class imbalance and bias in the dataset, enabling the model to more accurately capture dataset characteristics and adjust weights to maximize attention to the minority class. The fifth row of Table 4 shows that, based on the experiment in the fourth row, the randomization of mfreq[i] is removed, resulting in a 0.02 percentage point improvement in overall accuracy. Notably, this method achieves 90.65% and 46.43% accuracy for yes / no questions and other question types, respectively. Given the characteristics of the VQA-CPv1 dataset, question and answer distributions can be influenced by specific factors, such as the semantic characteristics of the questions and the complexity of the visual information. Therefore, the answer frequency distribution may exhibit certain regularities, and removing the randomization ensures that the model accurately exploits these regularities during training.

[0069] At the same time, on the VQA-CPv1 dataset, the t-SNE (t-distributed Stochastic Neighbor Embedding) visualization of the answers to different question types placed in the feature space in the first cycle of training for the second row model (the base model after adding bias detection) and the fourth row model (the model after adding dynamic loss) in Table 4 is shown in the figure below. Figure 6A t-SNE visualization of the answer feature space of a specific embodiment of the present application on the VQA-CP v1 dataset is shown. t-SNE is a technique for dimensionality reduction and visualization of high-dimensional data. One of its key features is its ability to preserve the local structure of the data within a smaller embedding space. This means that similar data points are typically clustered together in the embedding space, while dissimilar points are separated.

[0070] Table 4 Ablation test results of each component of the model on VQA-CP v1

[0071]

[0072] In the left image (a), different colors represent data points of different question types (e.g., "are," "do," "how many," "what color," and "where is the"). The densely colored areas reflect the distribution of these question types in the model's hidden layer outputs. In this image, we can see that questions of different categories overlap significantly and are not well separated. In the right image (b), the same colors represent the same category, but the data points are more dispersed, showing that questions of different categories are well separated. As training continues, the model in the right image will perform better than the one in the left image.

[0073] The ablation results of each model component on the VQA-CPv2 dataset are shown in Table 5. The scaling factor s is set to 8. The first row of Table 5 shows the experimental results of the base model UpDn on this dataset. By adding a deviation detection branch, further replacing it with a cosine dynamic loss, further adopting the dynamic margin proposed in this application, and further eliminating random processing, the overall accuracy is improved by 0.43%, 15.91%, 16.76%, and 16.31% respectively compared to the base model UpDn. It is worth noting that removing the randomization of mfreq[i] on the VQA-CPv2 dataset actually reduces performance by 0.45 percentage points. The VQA-CPv2 dataset has more complex and diverse question types and semantic contexts than the VQA-CPv1 dataset, which requires the model to have stronger generalization capabilities to handle various situations. Therefore, removing the randomization process may cause the model to rely on answer frequency information and ignore other features and semantic information. In addition, when dealing with other types of questions, the method only has an accuracy of 42.76%, which is much lower than the basic model (42.76% vs 43.42%).

[0074] Table 5 Ablation test results of each component of the model on VQA-CP v2

[0075]

[0076] The answer feature space after the initial training cycle on the VQA-CPv2 dataset is also visualized using t-SNE visualization technology. This visualization method not only shows the distribution of answer features of the base model and the model of this application when processing specific question types, but also provides an intuitive comparison to show the impact of adding dynamic loss on model features. Figure 7 The answer feature space of a specific embodiment of the present application is shown in a t-SNE visualization diagram on the VQA-CP v2 dataset. The left figure (a) shows the distribution of five question types (including "are", "are they", "how many", "is this", and "what color") in the answer feature space. These question types were selected as representative questions, and they cover a range from simple yes-or-no questions to more complex questions requiring counting and color recognition. In this figure, you can see how the basic model maps data points into the feature space after fusing features, while the right figure (b) shows how the optimized model of the present application processes these features.

[0077] It can be clearly seen that the dynamic loss makes the answer feature vectors more dispersed, which means that the model can more effectively distinguish different answer embeddings and avoid overlap between different answers. In addition, the proximity of similar answer embeddings in cosine space is also strengthened, thereby improving the model's ability to perceive answer similarity. This is because in cosine space, the angular relationship between vectors reflects their similarity, allowing the model to more accurately identify and distinguish semantically similar answers. This is particularly important for visual question answering tasks, because semantic understanding of the answers is key to solving the problem. This feature space optimization significantly improves the accuracy and robustness of the model in handling complex visual question answering tasks.

[0078] In order to more intuitively demonstrate the performance of the model in this application, the performance of the model in this application is compared with the baseline model AdaVQA on the VQA-CPv2 dataset.

[0079] First, this application selects five specific question types: "what", "what color", "what color is", "what color is the" and "what color are the". The prediction accuracy results are shown in Figure 8 The accuracy graph of the model of a specific embodiment of this application on VQA-CP v2 for five specific question types. Figure 8As shown in Figure 2, on these five question types, the baseline model AdaVQA achieved accuracies of 32.75%, 60.14%, 67.69%, 64.10%, and 64.61%, respectively, while the model trained in this application achieved accuracies of 34.56%, 68.38%, 74.25%, 65.68%, and 65.66%, respectively. Figure 8 As can be seen from the results, the model in this application performs better than AdaVQA on most question types.

[0080] Some typical examples are also selected for analysis in this application (see Figure 9 ) In which, the distribution of answers in the dataset for the question is displayed directly below the question. The first column on the right represents the true answer (Ground Truth, GT), the second column represents the answer predicted by the AdaVQA model, and the third column represents the answer predicted by the model of this application. The bounding boxes in the examples indicate the areas with the highest scores in the attention maps of each model. In the first image-question pair, the frequency of answers to "What color is the flower?" in the dataset shows that the frequency of the answer "pink" is significantly higher than that of "pink and yellow". The AdaVQA model is affected by this and predicts "pink". However, the model of this application dynamically adjusts the weights of high- and low-frequency samples. Even though "pink and yellow" is a low-frequency answer in the dataset, the model still locates the key area and correctly predicts the answer. In the second image-question pair, "2" and "4" are high-frequency answers in the dataset. Neither model predicted them, but AdaVQA chose the higher-frequency answer "8", while the model of this application chose the lower-frequency answer "5". From the attention maps predicted by the two models, it can be seen that the AdaVQA model locates a wide object area and is slightly affected by the bias of the data set, predicting the wrong answer. According to the attention map of the model of this application, it successfully identified the keyword "pens" in the question text and located the key area for answering the question. This shows that the model of this application can not only effectively remove bias, but also has certain robust performance, and the reasoning process is more reasonable. In addition, a common sample visualization was selected, that is, both models predict frequent answers in the data set. However, the model of this application has stronger robustness than the AdaVQA baseline model and can still make correct predictions.

[0081] Finally, this application selected three types of questions: "what color", "how many" and "are all", and visualized all the results predicted by the two models (see Figure 10A specific embodiment of the present application is shown in the figure comparing the answer feature distribution of different models in the cosine space). The color of the circle in each sub-figure represents all the prediction results of the model. Since there are many labels for the prediction results, the top 40 most frequently predicted labels are selected and divided into two columns and displayed in the legend on the right. Among them, (a) and (b) respectively represent the prediction results of the AdaVQA model and the model of the present application on the "whatcolor" question type in the cosine space; (c) and (d) respectively represent the prediction results of the AdaVQA model and the model of the present application on the "how many" question type in the cosine space. (e) and (f) respectively represent the prediction results of the AdaVQA model and the model of the present application on the "are all" question type in the cosine space.

[0082] As shown in Figure (a), the AdaVQA model generally tends to predict "pink," indicating that AdaVQA is susceptible to dataset bias. However, our model predicts "pink" much less frequently than AdaVQA, as shown in Figure (b). This fully demonstrates the effectiveness of dynamically adjusting sample weights. In the "what color" question type involving counting, the AdaVQA model also predicts "2" and "4" much more often than our model, as shown in Figures (c) and (d). This demonstrates that while the AdaVQA model has some ability to address dataset bias, it is insufficient for more complex situations. Furthermore, in cosine space, high-frequency answers have a wider range and therefore cover more samples. This leaves less room for low-frequency answers, and thus a smaller reasoning space. For the "are all" question type involving yes or no, the AdaVQA model generates 13 possible answers. While "both" accounts for a large proportion, the other twelve responses are significantly different from the "are all" question. On the contrary, although the model of this application only gives three answers, namely "no", "yes" and "both", it is not disturbed by other answers in the dataset. This shows that the model of this application more accurately captures the core semantics of the "are all" question type, which is in sharp contrast to the common phenomenon of scattered answers when traditional models deal with such questions.

[0083] This application adopts a method based on dynamic loss adjustment, with the aim of reducing or eliminating the impact of dataset bias during training. The model identifies dataset bias in samples by introducing a question-only branch and proposes a loss function based on answer frequency and bias detection. By dynamically adjusting the training weights of low-frequency and high-frequency samples, the bias problem in the visual question answering task is effectively improved, showing excellent performance. The above experimental results show that this weight adjustment strategy not only improves the model's sensitivity to rare samples, but also maintains the ability to efficiently process common samples, thereby achieving more balanced performance.

[0084] Figure 11 The framework diagram of a visual question answering model training system based on dynamic loss debiasing of a specific embodiment of the present application is shown, and the system includes a dynamic margin loss calculation unit 1101, a loss function fusion unit 1102 and a model training optimization unit 1103. Among them, the dynamic margin loss calculation unit 1101 is configured to dynamically adjust the margin parameters of the loss function according to the answer frequency of the training data and the sample deviation detection results, wherein the answer frequency generates a dynamic margin inversely proportional to the answer frequency by counting the number of occurrences of each answer under a given question type, and the sample deviation detection generates a dynamic margin proportional to the degree of deviation based on the deviation confidence of the question text by introducing an independent deviation detection branch; the loss function fusion unit 1102 is configured to weightedly fuse the dynamic margin based on the answer frequency with the dynamic margin based on the deviation detection to form the final dynamic margin; the model training optimization unit 1103 is configured to use the fused dynamic margin loss function to optimize the visual question answering model, and perform joint training in combination with the cross entropy loss and the loss of the deviation detection branch. Each unit of the system can execute the aforementioned Figure 2 The specific steps of the method.

[0085] Reference below Figure 12 , which shows a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. Figure 12 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0086] like Figure 12 As shown, the computer system includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage unit 1208 into a random access memory (RAM) 1203. Various programs and data required for the operation of the system 1200 are also stored in the RAM 1203. The CPU 1201, ROM 1202, and RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0087] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, and the like; an output section 1207 including a liquid crystal display (LCD) and speakers; a storage section 1208 including a hard disk; and a communication section 1209 including a network interface card such as a LAN card or a modem. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. Removable media 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1210 as needed, so that computer programs read therefrom can be installed into the storage section 1208 as needed.

[0088] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1209, and / or installed from the removable medium 1211. When the computer program is executed by the central processing unit (CPU) 1201, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0089] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0090] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0091] The modules described in the embodiments of the present application may be implemented in software or hardware.

[0092] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist independently and not be assembled into the electronic device. The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: dynamically adjusts the margin parameter of the loss function according to the answer frequency of the training data and the sample deviation detection result, wherein the answer frequency generates a dynamic margin inversely proportional to the answer frequency by counting the number of occurrences of each answer under a given question type, and the sample deviation detection generates a dynamic margin proportional to the degree of deviation by introducing an independent deviation detection branch and predicting the deviation confidence based on the question text; the dynamic margin based on the answer frequency is weightedly fused with the dynamic margin based on the deviation detection to form the final dynamic margin; the fused dynamic margin loss function is used to optimize the visual question answering model, and the cross entropy loss and the loss of the deviation detection branch are combined for joint training.

[0093] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A visual question answering model training method based on dynamic loss debiasing, characterized in that: include: S1: Dynamically adjust the margin parameter of the loss function based on the answer frequency of the training data and the sample deviation detection results, wherein the answer frequency is generated by counting the number of occurrences of each answer under a given question type, and a dynamic margin is generated that is inversely proportional to the answer frequency. The sample deviation detection generates a dynamic margin that is proportional to the degree of deviation by introducing an independent deviation detection branch and predicting the deviation confidence based on the question text; S2: weighted fusion of the dynamic margin based on the answer frequency and the dynamic margin based on deviation detection to form a final dynamic margin; S3: Use the fused dynamic margin loss function to optimize the visual question answering model, and combine the cross entropy loss and the loss of the bias detection branch for joint training.

2. The visual question answering model training method based on dynamic loss debiasing according to claim 1, characterized in that The dynamic margin calculation based on the answer frequency is specifically: in, Represents the answer a for a given question type in the training set i Frequency of occurrence, m freq [i] represents the dynamic margin of the answer ai based on the given question type, ∈ is a hyperparameter to avoid computational overflow, and |Ω| represents the full set of the answer space.

3. The visual question answering model training method based on dynamic loss debiasing according to claim 2, characterized in that: Under the given question type, for each answer a i , using a randomized version of the dynamic margin, and finally a dynamic margin based on the answer frequency Where N is the Gaussian distribution and σ is the standard deviation.

4. The visual question answering model training method based on dynamic loss debiasing according to claim 3 is characterized in that The deviation detection branch is a unimodal branch that only uses the question text, and the output is converted into deviation confidence through the Softmax function.

5. The visual question answering model training method based on dynamic loss debiasing according to claim 4, characterized in that: The formula for the dynamic margin of the sample deviation detection is m q [i]=softmax(f q / τ), τ represents the temperature coefficient, f q is the confidence of the deviation detection branch prediction, f q =e q (f bias ), e q is the classification head for prediction, f bias is the predicted answer distribution of the dataset bias captured by the bias branch according to the question information, and f bias =f(QW), where Q represents the question text and W is a learnable mapping parameter.

6. The visual question answering model training method based on dynamic loss debiasing according to claim 4, characterized in that In S2, the dynamic margin based on the answer frequency and the dynamic margin based on deviation detection are weighted and fused to form the final dynamic margin m comb [gt] = βm ran [gt]+(1—β)m q [gt], gt represents the index of the true label of the sample, β is a hyperparameter used to measure the two margins, β is set to 1 at the beginning of training and gradually decays linearly with the training rounds.

7. The visual question answering model training method based on dynamic loss debiasing according to claim 1, characterized in that The total loss function of the visual question answering model is: L total =L fre +L vqa +L q +L comb , where L fre is the loss function based on answer frequency, L comb is the adjusted cosine loss, L vqa is the cross entropy loss of the main branch of visual question answering, L q is the loss of the deviation detection branch.

8. The visual question answering model training method based on dynamic loss debiasing according to claim 7, characterized in that: Among them, a i is the weight factor of category i, scos() is the cosine similarity function, θ i represents the angle representation of the feature vector of category i, N represents the total number of training samples, t i represents the true label of the i-th sample, σ() is the sigmoid function, f(v i ,q i ) is the neural network model, v i is the image feature input, q i is the question text feature input.

9. A computer-readable storage medium having one or more computer programs stored thereon, characterized in that: When the one or more computer programs are executed by a computer processor, the method according to any one of claims 1 to 8 is implemented.

10. A visual question answering model training system based on dynamic loss debiasing, characterized in that: include: A dynamic margin loss calculation unit is configured to dynamically adjust the margin parameter of the loss function based on the answer frequency of the training data and the sample bias detection result, wherein the answer frequency is generated by counting the number of occurrences of each answer under a given question type to generate a dynamic margin that is inversely proportional to the answer frequency, and the sample bias detection is generated by introducing an independent bias detection branch to generate a dynamic margin that is proportional to the degree of bias based on the confidence level of the bias predicted by the question text; A loss function fusion unit configured to weightedly fuse the dynamic margin based on the answer frequency and the dynamic margin based on deviation detection to form a final dynamic margin; Model training optimization unit: Configured to optimize the visual question answering model using the fused dynamic margin loss function and perform joint training with the cross entropy loss and the loss of the bias detection branch.

Images