Training of visual question answering models, methods and devices for processing visual question answering tasks

Abstract

This disclosure relates to the field of machine learning technology, specifically to a training method and apparatus for a visual question-answering model, a visual question-answering task processing method and apparatus, a computer-readable storage medium, and an electronic device. The training method for the visual question-answering model includes: acquiring initial training data, which includes image and text input data and corresponding ground truth answers, wherein the image and text input data includes image feature data and text feature data; inputting the image and text input data into a first initial model to obtain a first reference answer; filtering the initial training data based on the first reference answer and the ground truth answers to obtain first target training data; and updating a second initial model using the first reference answer and the corresponding ground truth answers of the first target training data to obtain a visual question-answering model; wherein the model architecture of the first initial model and the second initial model is identical. The technical solution of this disclosure improves the processing accuracy of the obtained visual question-answering model.

Description

Technical Field

[0001] This disclosure relates to the field of machine learning technology, and more specifically, to a training method and apparatus for a visual question answering model, a visual question answering task processing method and apparatus, a computer-readable storage medium, and an electronic device. Background Technology

[0002] Visual question answering models have made great strides in the field of multimodal learning, but current visual question answering methods all require accurate data labels to build a complete dataset.

[0003] However, building a labeled dataset for a visual question answering model requires a huge amount of manpower and time, and the labeled data inevitably contains noise, resulting in low processing accuracy of the trained visual question answering model. Summary of the Invention

[0004] The purpose of this disclosure is to provide a training method for a visual question answering model, a training device for a visual question answering model, a method for processing visual question answering tasks, a device for processing visual question answering tasks, a computer-readable medium, and an electronic device, thereby improving the processing accuracy of the obtained visual question answering model to at least a certain extent.

[0005] According to a first aspect of this disclosure, a method for training a visual question-answering model is provided, comprising: acquiring initial training data, the initial training data including image and text input data and corresponding ground truth answers, wherein the image and text input data includes image feature data and text feature data; inputting the image and text input data into a first initial model to obtain a first reference answer; filtering the initial training data based on the first reference answer and the ground truth answers to obtain first target training data; updating a second initial model using the first reference answer and the corresponding ground truth answers of the first target training data to obtain a visual question-answering model; wherein the model architecture of the first initial model and the second initial model is consistent.

[0006] According to a second aspect of this disclosure, a training apparatus for a visual question-answering model is provided, comprising: a data acquisition module for acquiring initial training data, the initial training data including image and text input data and corresponding ground truth answers, wherein the image and text input data includes image feature data and text feature data; a data processing module for inputting the image and text input data into a first initial model to obtain a first reference answer; a data filtering module for filtering the initial training data based on the first reference answer and the ground truth answers to obtain first target training data; and a model updating module for updating a second initial model using the first reference answer and the corresponding ground truth answers of the first target training data to obtain a visual question-answering model; wherein the model architecture of the first initial model and the second initial model is identical.

[0007] According to a third aspect of this disclosure, a visual question answering task processing method is provided, comprising: obtaining reference image features and corresponding reference text features corresponding to the image to be asked; inputting the reference image features and reference text features into a visual question answering model to obtain a target answer; wherein the visual question answering model can be obtained according to the training method of the visual question answering model.

[0008] According to a fourth aspect of this disclosure, a visual question-answering task processing apparatus is provided, comprising: a feature acquisition module for acquiring reference image features and corresponding reference text features corresponding to the image to be asked; and a task processing module for inputting the reference image features and reference text features into a visual question-answering model to obtain a target answer; wherein the visual question-answering model is obtained according to a training method for a visual question-answering model.

[0009] According to a fifth aspect of this disclosure, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0010] According to a sixth aspect of this disclosure, an electronic device is provided, characterized in that it includes: one or more processors; and a memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method described above.

[0011] One embodiment of the present disclosure provides a training method for a visual question answering model. On the one hand, a first initial model is used to filter initial training data to obtain first target training data, thereby improving the accuracy of the training data of the visual question answering model and making the obtained visual question answering model more accurate. On the other hand, the parameters of a second initial model are updated by using the first reference answer output by the first initial model to obtain a visual question answering model, thereby further improving the noise resistance of the obtained visual question answering model and improving the accuracy of the visual question answering model.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0014] Figure 1 A schematic diagram of an exemplary system architecture to which embodiments of the present disclosure may be applied is shown;

[0015] Figure 2 The flowchart schematically illustrates a training method for a visual question-answering model in an exemplary embodiment of the present disclosure;

[0016] Figure 3 This schematic diagram illustrates an architecture of a visual question-answering model in an exemplary embodiment of the present disclosure.

[0017] Figure 4 The diagram schematically illustrates a partitioning diagram of an initial sub-model in an exemplary embodiment of the present disclosure;

[0018] Figure 5 A flowchart illustrating an exemplary embodiment of the present disclosure is provided.

[0019] Figure 6 The diagram illustrates the data flow of a training method for a visual question-answering model in an exemplary embodiment of the present disclosure.

[0020] Figure 7 This schematically illustrates a flowchart of a visual question-answering task processing method according to an exemplary embodiment of the present disclosure;

[0021] Figure 8 This schematically illustrates a data flow diagram of a visual question-answering task processing method in an exemplary embodiment of the present disclosure;

[0022] Figure 9 This schematic diagram illustrates the composition of a visual question-answering task processing apparatus in an exemplary embodiment of the present disclosure.

[0023] Figure 10 This schematic diagram illustrates the composition of a training apparatus for a visual question-answering model in an exemplary embodiment of the present disclosure.

[0024] Figure 11 A schematic diagram of an electronic device to which embodiments of the present disclosure may be applied is shown. Detailed Implementation

[0025] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0026] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0027] Visual question answering is a practical machine learning task that requires an AI model to output an answer to a visual question about an image. The challenge lies in its multi-tasking and open-ended nature, involving the simultaneous resolution of multiple technical research problems in computer vision and natural language understanding. Advances in this task will enable a wide range of applications for multimodal machine learning, from assisting the blind and visually impaired to communicating with robots to enhance users' visual experience and external knowledge.

[0028] Existing visual question answering systems suffer from the following problems: without a large-scale initial training dataset of high-quality visual images, questions, and answers in diverse triples, it is difficult to train an effective and stable visual question answering model.

[0029] However, preparing a complete dataset is both time-consuming and cumbersome. Current research in visual question answering primarily focuses on designing complex models, rather than constructing training data needed for real-world scenarios. Therefore, how to enable robust training of models with lower-quality annotations is a pressing issue that needs to be addressed.

[0030] To address the aforementioned shortcomings, this disclosure provides a training method for a visual question-answering model. Figure 1 A schematic diagram of a system architecture for implementing the training method of the above-described visual question-answering model is shown. This system architecture 100 may include a terminal 110 and a server 120. The terminal 110 may be a smartphone, tablet, desktop computer, laptop, or other terminal device. The server 120 generally refers to the backend system providing visual question-answering related services in this exemplary embodiment, and may be a single server or a cluster of multiple servers. The terminal 110 and the server 120 can be connected via a wired or wireless communication link for data interaction.

[0031] In one implementation, the training method for the visual question-answering model described above can be executed by the terminal 110. For example, a user uses the terminal 110 to obtain initial training data, which includes image and text input data and the corresponding ground truth answers. The image and text input data includes image feature data and text feature data. The terminal 110 first inputs the image and text input data into a first initial model to obtain a first reference answer. Then, based on the first reference answer and the ground truth answer, the initial training data is filtered to obtain first target training data. Finally, the first reference answer and the corresponding ground truth answer of the first target training data are used to update the second initial model to obtain the visual question-answering model. The model architecture of the first initial model and the second initial model is consistent.

[0032] In one implementation, the server 120 can execute the above-described training method for the visual question-answering model. For example, a user uses a terminal 110 to obtain initial training data, which includes image and text input data and the corresponding ground truth answers. The image and text input data includes image feature data and text feature data. The terminal 110 then uploads the initial training data to the server 120. The server 120 first inputs the image and text input data into a first initial model to obtain a first reference answer. Then, based on the first reference answer and the ground truth answer, it filters the initial training data to obtain first target training data. Finally, it uses the first reference answer and the corresponding ground truth answer of the first target training data to update a second initial model to obtain a visual question-answering model, and then returns the visual question-answering model to the terminal 110.

[0033] As can be seen from the above, the execution subject of the training method of the visual question answering model in this exemplary embodiment can be the aforementioned terminal 110 or server 120, and this disclosure does not limit it in this regard.

[0034] The following is combined with Figure 2 The training method of the visual question-answering model in this exemplary embodiment will be described. Figure 2 An exemplary flow of the training method for the visual question answering model is shown, which may include steps S210 to S240.

[0035] refer to Figure 2 In step S210, initial training data is obtained, which includes image and text input data and the true answer corresponding to the image and text input data. The image and text input data includes image feature data and text feature data.

[0036] In one example embodiment of this disclosure, the processor may first acquire initial training data, wherein the initial training data includes image and text input data and the true answer corresponding to the image and text input data, wherein the image and text input data may include image feature data and text feature data.

[0037] Specifically, the processor can first obtain the initial image, the initial text question, and the truth answer corresponding to the initial text question. Then, it can perform feature extraction on the initial image and the initial text question to obtain image feature data and text feature data.

[0038] In one example implementation, Faster-RCNN (Regions with CNN features) can be used to extract features from the initial image to obtain image feature data. In this example implementation, Faster-RCNN is used to implement a bottom-up attention model. By setting a threshold to allow overlap of interest boxes, the image content can be understood more effectively, resulting in more accurate image feature data.

[0039] In one example implementation, a text feature extraction model can be used to extract features from the initial question text. This model may include two LSTM (Long Short-Term Memory) networks, one of which is an LSTM with a top-down attention mechanism that helps reweight image features. After feature extraction, a GRU (Gated Recurrent Unit) module is used for sequence encoding.

[0040] In step S220, the image and text input data is input into the first initial model to obtain the first reference answer.

[0041] In one example embodiment of this disclosure, after obtaining the above-mentioned graphic input data, the graphic input data can be input into the first initial model to obtain the first reference answer.

[0042] The first initial model and the second initial model have the same framework structure. The model parameters in the first initial model and the second initial model can be the same or different. Preferably, the parameters in the first initial model and the second initial model are different. During training, the second initial model can absorb the noise resistance of the first initial model, so that the accuracy of the trained visual question answering model is higher.

[0043] In this example implementation, the first and second initial models mentioned above can be model structures of UpDn and GRU, and the initial training data is D = {I} i Q i ,a i} N It contains N images I i Question Q i Answer a i right.

[0044] Specifically, refer to Figure 3 As shown, the first and second initial models mentioned above may include a top-down attention LSTM module 310, an attend (weighted) module 320, a language LSTM module 330, and a loss function layer 340.

[0045] Specifically, the input to the top-down attention LSTM module at time step t is:

[0046]

[0047] in, This is the output of the previous round of the language LSTM module. W is the average value of the image feature data. e It is the embedding matrix of the question-and-answer text, ∏ t These are the one-hot encoded words of the current round's input. The top-down attention LSTM module provides the current language model's text information, image information, summary, and descriptive information.

[0048] Furthermore, weights can be assigned to each feature data.

[0049]

[0050] And by using a softmax function, we obtain: a t =softmax(a t )

[0051] Finally, the image weighted features are obtained:

[0052]

[0053] On the other hand, a bottom-up LSTM network is used to calculate the probability distribution, specifically by combining image weighted features and... The inputs to the language LSTM module are:

[0054]

[0055] At this point, the probability distribution of the predicted word (i.e., the reference answer) at time t is:

[0056]

[0057] The loss functions for the first and second initial models mentioned above are:

[0058]

[0059] Among them, y i p represents the label of sample i, with 1 for positive and 0 for negative. i This represents the probability that sample i is predicted to be of the positive class.

[0060] It should be noted that the above description of the specific framework of the first initial model and the second initial model is exemplary, and no detailed limitation is made on the specific architecture of the first initial model and the second initial model in this disclosure.

[0061] In step S230, the initial training data is filtered based on the first reference answer and the true answer to obtain the first target training data.

[0062] In this example implementation, the method may include steps S310 and S330.

[0063] In step S310, the loss value between the first reference answer and the true answer is calculated.

[0064] In this example implementation, the loss value between the first reference answer and the true answer can be determined using the loss function determined above. Alternatively, the similarity between the first reference answer and the true answer can be determined first, and the loss value can be obtained by subtracting the similarity from 1. The calculation of the loss value can also be customized according to user needs, and no specific limitation is made in this example implementation.

[0065] In step S320, the initial training data corresponding to the first reference answer with a loss value less than a preset threshold is used as the first target training data.

[0066] After obtaining the loss values ​​corresponding to each first reference, a preset threshold can be determined. The preset threshold can be 0.2, 0.3, etc., or it can be customized according to user needs. Then, the above loss values ​​are compared with the preset thresholds respectively, and the initial training data corresponding to the first reference answer with a loss value less than the preset threshold is used as the first target training data.

[0067] Using the loss value to filter the initial training data and obtain the first target data can reduce noise during training, improve the accuracy of the first target training data, and enable the trained visual question answering model to have higher processing accuracy.

[0068] In one example embodiment of this disclosure, obtaining the first target training data may further include steps S410 to S430.

[0069] In step S410, the initial training data is divided into multiple groups of initial sub-data;

[0070] In this example implementation, refer to Figure 4 As shown, the initial training data can be divided into multiple initial sub-data sets. The number of image feature data and text feature data in each initial sub-data set can be one or more, such as three, five, etc. The number of initial training data in each initial sub-data set can be the same or different. No specific limitation is made in this example implementation.

[0071] In step S420, the target sub-data is obtained by filtering each initial sub-data based on the first reference answer and the true answer;

[0072] After grouping, the first reference answer and the truth answer are used to filter each initial sub-data to obtain multiple target sub-data. The specific filtering process can be referred to steps S310 to S320, and is not specifically limited in this example implementation.

[0073] In step S430, multiple sets of target sub-data are used as the first target training data.

[0074] After obtaining multiple target sub-data, these multiple target sub-data can be used as the training data for the first target mentioned above. The initial training data can be divided into multiple groups for filtering. Multiple groups can be filtered simultaneously, which can increase or decrease the filtering rate. At the same time, when the processor is busy, each initial sub-data can be filtered sequentially, which can reduce the processing pressure on the processor. The initial training data can be filtered under narrow bandwidth conditions.

[0075] In step S240, the second initial model is updated using the first reference answer and the corresponding true answer of the first target training data to obtain the visual question answering model.

[0076] In this example implementation, after obtaining the first target training data, the second initial model can be updated using the first reference answer and the true answer corresponding to the first reference answer in the target initial training data to obtain the visual question answering model.

[0077] Specifically, the first gradient value of backpropagation can be determined based on the first reference answer and the true answer corresponding to the first reference answer in the training data of the first target mentioned above. Then, the parameters in the second initial model are updated using the calculated first gradient value to obtain the visual question answering model.

[0078] In one exemplary embodiment of this disclosure, reference is made to Figure 5 As shown, the training method for the above-mentioned visual question answering model may further include steps S250 to S270.

[0079] In step S250, the image and text input data are input into the second initial model to obtain the second reference answer.

[0080] In one example embodiment of this disclosure, after obtaining the above-mentioned graphic input data, the above-mentioned graphic input data can be input into the above-mentioned second initial model to obtain the above-mentioned second reference answer.

[0081] The first initial model and the second initial model have the same framework structure. The model parameters in the first initial model and the second initial model can be the same or different. Preferably, the parameters in the first initial model and the second initial model are different. During training, the second initial model can absorb the noise resistance of the first initial model, so that the accuracy of the trained visual question answering model is higher.

[0082] In step S260, the initial training data is filtered based on the second reference answer and the true answer to obtain the second target training data.

[0083] In this example implementation, the method may include steps S510 and S530.

[0084] In step S510, the loss value between the second reference answer and the true answer is calculated.

[0085] In this example implementation, the loss value between the second reference answer and the true answer can be determined using the loss function determined above. Alternatively, the similarity between the second reference answer and the true answer can be determined first, and the loss value can be obtained by subtracting the similarity from 1. The calculation of the loss value can also be customized according to user needs, and no specific limitation is made in this example implementation.

[0086] In step S520, the initial training data corresponding to the second reference answer with a loss value less than a preset threshold is used as the second target training data.

[0087] After obtaining the loss values ​​corresponding to each second reference, a preset threshold can be determined. The preset threshold can be 0.2, 0.3, etc., or it can be customized according to user needs. Then, the above loss values ​​are compared with the preset thresholds respectively, and the initial training data corresponding to the second reference answer with a loss value less than the preset threshold is used as the above second target training data.

[0088] Using the loss value to filter the initial training data and obtain the second target data can reduce noise during training, improve the accuracy of the second target training data, and enable the trained visual question answering model to have higher processing accuracy.

[0089] In one example embodiment of this disclosure, obtaining the training data for the second target may further include steps S610 to S630.

[0090] In step S610, the initial training data is divided into multiple groups of initial sub-data;

[0091] In this example implementation, refer to Figure 5 As shown, the initial training data can be divided into multiple initial sub-data sets. The number of image feature data and text feature data in each initial sub-data set can be one or more, such as three, five, etc. The number of initial training data in each initial sub-data set can be the same or different. No specific limitation is made in this example implementation.

[0092] In step S620, the target sub-data is obtained by filtering each initial sub-data based on the second reference answer and the true answer;

[0093] After grouping, the initial sub-data are filtered using the second reference answer and the truth answer to obtain multiple target sub-data. The specific filtering process can be referred to steps S510 to S520, and is not specifically limited in this example implementation.

[0094] In step S630, multiple sets of target sub-data are used as training data for the second target.

[0095] After obtaining multiple target sub-data, these sub-data can be used as the training data for the second target. The initial training data can be divided into multiple groups for filtering. Multiple groups can be filtered simultaneously, which can increase or decrease the filtering rate. At the same time, when the processor is busy, each initial sub-data can be filtered sequentially, which can reduce the processing pressure on the processor. The initial training data can be filtered under narrow bandwidth conditions.

[0096] In step S270, the first initial model is updated using the second reference answer and the corresponding true answer from the second target training data to obtain the visual question answering model.

[0097] In this example implementation, after obtaining the second target training data, the second initial model can be updated using the second reference answer and the truth answer corresponding to the second reference answer in the second target training data to obtain the visual question answering model.

[0098] Specifically, the second gradient value of backpropagation can be determined based on the second reference answer and the true answer corresponding to the second reference answer in the training data of the second objective mentioned above. Then, the calculated second gradient value is used to update the parameters in the first initial model to obtain the visual question answering model.

[0099] In this example implementation, refer to Figure 6As shown, the output of the first initial model and the true answer can be used to update the second initial model, and the output of the second initial model and the true answer can be used to update the first initial model to obtain two visual question answering models. Either visual question answering model can complete the visual question answering task.

[0100] For example, the initial sub-data includes three initial training data sets, which can be fed into the first initial model. The loss values ​​of the three initial training data sets are calculated, and two first target sub-data sets are selected. The gradient values ​​of the backpropagation for the first target sub-data sets are calculated, and these gradient values ​​are used to train the second initial model. Similarly, the initial sub-data can be fed into the second initial model. The loss values ​​of the three initial training data sets are calculated, and two second target sub-data sets are selected. The gradient values ​​of the backpropagation for the second target sub-data sets are calculated, and these gradient values ​​are used to train the first initial model. After training all the initial sub-data sets, two visual question answering models are obtained, both of which can perform visual question answering tasks with relatively high accuracy.

[0101] In summary, in this exemplary embodiment, on the one hand, by filtering the initial training data using the first initial model to obtain the first target training data, the accuracy of the initial training data is improved, resulting in a higher accuracy visual question-answering model. On the other hand, by updating the parameters of the second initial model using the first reference answer output by the first initial model to obtain the visual question-answering model, the noise resistance of the obtained visual question-answering model is further improved, thus enhancing its accuracy. Furthermore, using the loss value to filter the initial training data to obtain the first target data reduces noise during training, improves the accuracy of the first target training data, and enables the trained visual question-answering model to achieve higher processing accuracy. Moreover, dividing the initial training data into multiple groups for filtering allows for simultaneous filtering of multiple groups, increasing or decreasing the filtering rate. Additionally, when the processor is busy, filtering each initial sub-data sequentially reduces the processor's processing load, allowing the filtering of the initial training data to be completed even with limited bandwidth.

[0102] Furthermore, refer to Figure 7 As shown, this disclosure also provides a visual question-answering task processing method, which can be performed by... Figure 1 The above-mentioned terminal can be executed, or it can be executed by... Figure 1 The server executes the process, for example, the user uses terminal 110 to obtain the reference image features and corresponding reference text features corresponding to the image to be asked, and simultaneously loads a visual question answering model, which is trained using the aforementioned visual model training method. Then, the aforementioned reference image features and reference text features are input into the visual question answering model to obtain the target answer, thus completing the processing of the aforementioned visual question answering task.

[0103] For example, after a user uses terminal 110 to obtain the reference image features and the corresponding reference text features of the image to be asked, the user uploads the reference image features and the corresponding reference text features of the image to be asked to server 120. Server 120 loads a visual question answering model, which is trained using the aforementioned visual model training method. Then, the visual question answering model is used to process the reference image features and the corresponding reference text features of the image to be asked to obtain the target answer.

[0104] The visual question answering task processing method may specifically include steps S710 to S720.

[0105] In step S710, the reference image features and the corresponding reference text features corresponding to the image to be questioned are obtained.

[0106] In this example implementation, refer to Figure 8 As shown, after obtaining the image to be asked and the corresponding question, the image to be asked can be input into the image feature extraction module to obtain reference image features, and the question can be input into the text feature extraction module to obtain reference text features.

[0107] The specific processes for image feature extraction and text feature extraction can be found in the training methods for visual question answering models, and will not be elaborated here.

[0108] In step S720, the reference image features and reference text features are input into the visual question answering model to obtain the target answer.

[0109] After obtaining the reference image features and reference text features, the reference image features and reference text features can be input into the visual question answering model to obtain the target answer.

[0110] The specific structure of the visual question answering model can be found in the training method of the visual question answering model, and will not be elaborated here.

[0111] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0112] Further reference Figure 9 As shown, this example embodiment also provides a training device 900 for a visual question-answering model, including a data acquisition module 910, a data processing module 920, a data filtering module 930, and a model update module 940. Wherein:

[0113] The data acquisition module 910 can be used to acquire initial training data, which includes image and text input data and the corresponding ground truth answers. The image and text input data includes image feature data and text feature data. The data processing module 920 can be used to input the image and text input data into a first initial model to obtain a first reference answer. The data filtering module 930 can be used to filter the initial training data based on the first reference answer and the ground truth answers to obtain first target training data. The model update module 940 can be used to update a second initial model with the first reference answer and the corresponding ground truth answers of the first target training data to obtain a visual question answering model. The model architectures of the first and second initial models are identical.

[0114] In one example implementation, the data acquisition module 910 can be configured to acquire an initial image, an initial question text, and the truth answer corresponding to the initial question text; and to perform feature extraction on multiple initial images and initial question texts to obtain image feature data and text feature data.

[0115] In one example implementation, the data filtering module 930 can be configured to calculate the loss value between the first reference answer and the true answer; and to use the initial training data corresponding to the first reference answer whose loss value is less than a preset threshold as the first target training data.

[0116] In another example implementation, the data filtering module 930 can be configured to divide the initial training data into multiple initial sub-data; filter each initial sub-data based on the first reference answer and the true answer to obtain target sub-data; and use the multiple sets of target sub-data as the first target training data.

[0117] In one example implementation, the model update module 940 can be configured to update the second initial model using the second reference answer and the corresponding truth answer in each target sub-data to obtain a visual question answering model.

[0118] Further reference Figure 10 As shown, this example embodiment also provides a visual question-answering task processing device 1000, including a feature acquisition module 1010 and a task processing module 1020.

[0119] in:

[0120] The feature acquisition module 1010 can be used to acquire the reference image features and the corresponding reference text features corresponding to the image to be asked; the task processing module 1020 can be used to input the reference image features and the reference text features into the visual question answering model to obtain the target answer; wherein, the visual question answering model is obtained by the training method of the visual question answering model according to any one of claims 1-6.

[0121] The specific details of each module in the above-mentioned device have been described in detail in the method section of the implementation. For any undisclosed details, please refer to the implementation content of the method section, and therefore will not be repeated here.

[0122] Exemplary embodiments of this disclosure also provide an electronic device for performing the training method of the visual question-answering model described above. This electronic device may be the terminal 110 or the server 120 described above. Generally, the electronic device may include a processor and a memory, the memory for storing executable instructions of the processor, and the processor configured to perform the training method of the visual question-answering model by executing the executable instructions.

[0123] The following is based on Figure 11 Taking the mobile terminal 1100 as an example, the construction of this electronic device will be described by way of example. Those skilled in the art will understand that, apart from components specifically designed for mobile purposes, Figure 11 The structure can also be applied to fixed types of equipment.

[0124] like Figure 11 As shown, the mobile terminal 1100 may specifically include: a processor 1101, a memory 1102, a bus 1103, a mobile communication module 1104, an antenna 1, a wireless communication module 1105, an antenna 2, a display screen 1106, a camera module 1107, an audio module 1108, a power module 1109, and a sensor module 1110.

[0125] Processor 1101 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, an encoder, a decoder, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The training method for the visual question-answering model in this exemplary embodiment can be executed by an AP, GPU, or DSP. When the method involves neural network-related processing, it can be executed by an NPU.

[0126] The processor 1101 can be connected to the memory 1102 or other components via the bus 1103.

[0127] The memory 1102 can be used to store computer executable program code, which includes instructions. The processor 1101 executes various functional applications and data processing of the mobile terminal 1100 by running the instructions stored in the memory 1102. The memory 1102 can also store application data, such as images, videos, and other files.

[0128] The communication function of mobile terminal 1100 can be implemented through mobile communication module 1104, antenna 1, wireless communication module 1105, antenna 2, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 1104 can provide 2G, 3G, 4G, and 5G mobile communication solutions for mobile terminal 1100. Wireless communication module 1105 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for mobile terminal 1100.

[0129] The display screen 1106 is used to implement display functions, such as displaying the user interface, images, and videos. The camera module 1107 is used to implement shooting functions, such as capturing images and videos. The audio module 208 is used to implement audio functions, such as playing audio and capturing voice. The power module 209 is used to implement power management functions, such as charging the battery, powering the device, and monitoring the battery status. The sensor module 1110 may include a depth sensor 11101, a pressure sensor 11102, a gyroscope sensor 11103, a barometric pressure sensor 11104, etc., to implement corresponding sensing and detection functions.

[0130] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0131] Exemplary embodiments of this disclosure also provide a computer-readable storage medium having a program product stored thereon capable of implementing the methods described above in this specification. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0132] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0133] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0134] Furthermore, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0135] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0136] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for a visual question-answering model, characterized in that, The method comprises: obtaining initial training data, wherein the initial training data comprises image-text input data and a true value answer corresponding to the image-text input data, and the image-text input data comprises image feature data and text feature data; inputting the image-text input data into a first initial model to obtain a first reference answer; screening the initial training data based on the first reference answer and the true value answer to obtain first target training data; updating a second initial model based on the first reference answer of the first target training data and the corresponding true value answer to obtain the visual question answering model; wherein the model architecture of the first initial model and the second initial model is consistent; The method further comprises: inputting the image-text input data into the second initial model to obtain a second reference answer; screening the initial training data based on the second reference answer and the true value answer to obtain second target training data; updating the first initial model based on the second reference answer of the second target training data and the corresponding true value answer to obtain the visual question answering model.

2. The method of claim 1, wherein, The screening of the initial training data based on the first reference answer and the true value answer to obtain the first target training data comprises: calculating a loss value of the first reference answer and the true value answer; regarding the initial training data corresponding to the first reference answer with a loss value less than a preset threshold as the first target training data.

3. The method of claim 1, wherein, The screening of the initial training data based on the first reference answer and the true value answer to obtain the first target training data comprises: dividing the initial training data into multiple groups of initial sub-data; screening each of the initial sub-data based on the first reference answer and the true value answer to obtain target sub-data; regarding the multiple groups of target sub-data as the first target training data.

4. The method of claim 3, wherein, The updating of the second initial model based on the first reference answer of the first target training data and the corresponding true value answer to obtain the visual question answering model comprises: updating the second initial model based on the second reference answer in each of the target sub-data and the corresponding true value answer to obtain the visual question answering model.

5. The method of claim 1, wherein, The obtaining of the initial training data comprises: obtaining an initial image, an initial question text, and a true value answer corresponding to the initial question text; extracting features of the initial image and the initial question text to obtain the image feature data and the text feature data.

6. A visual question answering task processing method, characterized by, The method comprises: obtaining reference image features corresponding to a to-be-asked image and corresponding reference text features; inputting the reference image features and the reference text features into a visual question answering model to obtain a target answer; wherein the visual question answering model can be obtained according to the training method of the visual question answering model in any one of claims 1 to 5.

7. A training device for a visual question-answering model, characterized in that, The method comprises: a data acquisition module configured to obtain initial training data, wherein the initial training data comprises image-text input data and a true value answer corresponding to the image-text input data, and the image-text input data comprises image feature data and text feature data; a data processing module configured to input the image-text input data into a first initial model to obtain a first reference answer; The data screening module is configured to screen the initial training data based on the first reference answer and the true value answer to obtain first target training data. The model updating module is configured to update the second initial model based on the first reference answer of the first target training data and the corresponding true value answer to obtain the visual question answering model. The first initial model and the second initial model have consistent model architectures. The apparatus is further configured to: input the image-text input data into the second initial model to obtain a second reference answer; screen the initial training data based on the second reference answer and the true value answer to obtain second target training data; update the first initial model based on the second reference answer of the second target training data and the corresponding true value answer to obtain the visual question answering model.

8. A visual question answering task processing apparatus characterized by comprising: The apparatus comprises: a feature acquisition module configured to acquire reference image features and corresponding reference text features corresponding to a to-be-asked image; a task processing module configured to input the reference image features and the reference text features into a visual question answering model to obtain a target answer; The visual question answering model is obtained according to the training method of the visual question answering model in any one of claims 1 to 5.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the method in any one of claims 1 to 6.

10. An electronic device, comprising: The apparatus comprises: one or more processors; and a memory configured to store one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method in any one of claims 1 to 6. The apparatus comprises: one or more processors; and a memory configured to store one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method in any one of claims 1 to 6.

Images