Training method, method and device for answering questions

Abstract

The application provides a training method of a question answering model, a question answering method and device, wherein the training method comprises the following steps: obtaining pre-training data, a third test question image and corresponding third test question text and third answer text; the pre-training data comprises first test question text and corresponding first answer text, and a second test question image and the position and text content of each text line thereof; an initial model is trained based on the pre-training data to obtain a pre-training model; the pre-testing question text and the predicted answer text corresponding to the third test question image are determined based on the pre-training model; and the model is fine-tuned based on the pre-testing question text and the predicted answer text corresponding to the third test question image, and the third test question text and the third answer text, so as to obtain a question answering model, which overcomes the defect that the current question answering scheme cannot answer questions containing images, effectively answers any type of question through the trained question answering model, guarantees the accuracy of question answering, and improves the efficiency of question answering.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a training method, answering method and apparatus for a question-answering model. Background Technology

[0002] With the popularization of the internet and the rise of online education, students can more easily access various learning resources and information. However, how to effectively utilize these resources to improve learning efficiency and outcomes has become an urgent problem to be solved. In this context, automatic answering technology has emerged, which can quickly and accurately provide answers to questions, thereby helping students better understand and master knowledge, and thus improve learning efficiency.

[0003] However, most current automatic question-answering technologies use a transformer-based natural language processing architecture. If the original question is an image, text recognition is performed first to obtain the question information. Then, the question information is input into the model to obtain the final answer text. Although this method achieves automatic question answering, it is only applicable to questions with plain text and cannot effectively answer questions containing images. Summary of the Invention

[0004] This invention provides a training method, answering method, and apparatus for a question-answering model, which addresses the shortcomings of existing technologies that can only automatically answer questions with plain text but cannot automatically answer questions containing images. The trained question-answering model enables effective answering of questions of any type, improving answering efficiency while ensuring accuracy.

[0005] This invention provides a method for training a question-answering model, comprising:

[0006] Acquire pre-training data and training data. The pre-training data includes a first test question text and its corresponding first answer text, as well as a second test question image and the position and text content of each text line therein. The training data includes a third test question image, as well as the third test question text and the third answer text corresponding to the third test question image.

[0007] Based on the pre-training data, the initial model is trained to obtain the pre-trained model;

[0008] Based on the pre-trained model, the predicted test question text and predicted answer text corresponding to the third test question image are determined;

[0009] Based on the predicted test question text and predicted answer text corresponding to the third test question image, as well as the third test question text and the third answer text, the pre-trained model is fine-tuned to obtain the answer model.

[0010] According to a training method for a question-answering model provided by the present invention, the step of determining the pre-test question text and predicted answer text corresponding to the third question image based on the pre-trained model includes:

[0011] Based on the pre-trained model, the pre-test question text corresponding to the third test question image is determined;

[0012] Based on the pre-trained model, the third test question image and its corresponding pre-test question text, the predicted answer text corresponding to the third test question image is determined.

[0013] According to a training method for a question-answering model provided by the present invention, the step of training an initial model based on the pre-training data to obtain a pre-trained model includes:

[0014] Based on the initial model, the predicted answer text corresponding to the first question text is determined;

[0015] Based on the predicted answer text corresponding to the first test question text and the first answer text, the initial model is trained to obtain the first pre-trained model;

[0016] Based on the first pre-trained model, the predicted position of each text line in the second test question image and the predicted content of each text line are determined.

[0017] Based on the position and content of each text line, as well as the predicted position and the predicted content, the first pre-trained model is trained to obtain the pre-trained model.

[0018] According to a training method for a question-answering model provided by the present invention, the method further includes fine-tuning the pre-trained model based on the predicted question text and predicted answer text corresponding to the third question image, as well as the third question text and the third answer text, to obtain a question-answering model.

[0019] Obtain the answer image, as well as the corresponding question text and answer result;

[0020] Based on the answering model, the pre-test question text corresponding to the answering image is determined;

[0021] Based on the answering model, the answering image and its corresponding predicted question text, the predicted answering result corresponding to the answering image is determined;

[0022] Based on the predicted test question text and predicted answer result corresponding to the answer image, as well as the test question text and the answer result, the answer model is fine-tuned to obtain the review model.

[0023] According to the training method of the question-answering model provided by the present invention, the initial model is constructed based on an image encoder and a cognitive model.

[0024] This invention also provides a method for answering questions, including:

[0025] Identify the image of the question to be answered;

[0026] Based on the question-answering model, the answer text corresponding to the question image is determined;

[0027] The answering model is determined based on the training method of the answering model described in any of the above.

[0028] According to a question-answering method provided by the present invention, determining the answer text corresponding to the question image based on a question-answering model includes:

[0029] Based on the answering model, determine the question image features corresponding to the question image;

[0030] Determine the test question prompt text, and determine the text features based on the test question prompt text;

[0031] Based on the answer model, the question image features, and the text features, the answer text corresponding to the question image is determined.

[0032] According to a question-answering method provided by the present invention, determining the answer text corresponding to the question image based on the question-answering model, the question image features, and the text features includes:

[0033] Based on the answering model, attention interaction is performed on the question image features and the text features to obtain the question text corresponding to the question image;

[0034] Based on the test question text, determine the answer prompt text, and based on the answer prompt text, determine the answer text features;

[0035] Based on the answering model, attention interaction is performed on the question image features and the answer text features to obtain the answer text corresponding to the question image.

[0036] According to the question-answering method provided by the present invention, it further includes:

[0037] Identify the answer images to be reviewed;

[0038] Based on the review model, the answer image features corresponding to the answer image are determined;

[0039] Determine the test question prompt text, and determine the text features based on the test question prompt text;

[0040] Based on the review model, the answer image features, and the text features, the answer result corresponding to the answer image is determined.

[0041] According to a question-answering method provided by the present invention, determining the question-answering result corresponding to the question-answering image based on the review model, the question-answering image features, and the text features includes:

[0042] Based on the review model, attention interaction is performed on the answer image features and the text features to obtain the test text corresponding to the answer image;

[0043] Based on the question text corresponding to the answer image, the review prompt text is determined, and the review text features are determined based on the review prompt text;

[0044] Based on the review model, attention interaction is performed on the features of the answer image and the features of the review text to obtain the answer result corresponding to the answer image.

[0045] The present invention also provides a training device for a question-answering model, comprising:

[0046] The acquisition unit is used to acquire pre-training data and training data. The pre-training data includes a first test question text and its corresponding first answer text, as well as a second test question image and the position and text content of each text line therein. The training data includes a third test question image, as well as the third test question text and the third answer text corresponding to the third test question image.

[0047] A pre-training unit is used to train an initial model based on the pre-training data to obtain a pre-trained model;

[0048] The prediction unit is used to determine the predicted test question text and the predicted answer text corresponding to the third test question image based on the pre-trained model.

[0049] The training unit is used to fine-tune the pre-trained model based on the predicted test question text and predicted answer text corresponding to the third test question image, as well as the third test question text and the third answer text, to obtain the answer model.

[0050] The present invention also provides a question-answering device, comprising:

[0051] The unit is used to determine the image of the question to be answered;

[0052] The answering unit is used to determine the answer text corresponding to the question image based on the answering model; the answering model is determined based on the training method of the answering model as described above.

[0053] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a training method or a question-answering method for a question-answering model as described above.

[0054] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method or a question-answering method for a question-answering model as described above.

[0055] The training method, answering method, and apparatus for the answering model provided by this invention improve the initial model's ability to understand and learn knowledge, answer questions, and predict the location and content of text in images through pre-training. The initial model is fine-tuned using training data during the training phase to better adapt it to answering tasks, enabling it to answer questions containing images. This overcomes the limitation of traditional answering schemes that cannot answer questions containing images. The trained answering model achieves effective answers to questions of any type, improving answering efficiency while ensuring accuracy. This benefits student learning efficiency and optimizes the user experience of related learning devices. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0057] Figure 1 This is a flowchart illustrating the training method of the question-answering model provided by the present invention;

[0058] Figure 2 This is a general framework diagram of the training method for the question-answering model provided by the present invention;

[0059] Figure 3 This is a flowchart illustrating the question-answering method provided by the present invention;

[0060] Figure 4 This is a flowchart illustrating the question-answering process provided by the present invention;

[0061] Figure 5 This is a flowchart illustrating the review process provided by the present invention;

[0062] Figure 6 This is a schematic diagram of the structure of the training device for the question-answering model provided by the present invention;

[0063] Figure 7This is a schematic diagram of the structure of the question-answering device provided by the present invention;

[0064] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0066] Currently, most math answering technologies use transformer-based natural language processing models. If the question is in image format, an OCR (Optical Character Recognition) system is first used to recognize the text and obtain the question information. Then, the question information is input into the model to obtain the final answer text.

[0067] However, the above approach only works for pure text questions, where the model's input and output are both text, without any understanding or processing of images. Therefore, the model cannot answer questions containing images, such as plane geometry, solid geometry, or statistical graphs. Furthermore, current question-answering technologies have limited ability to understand natural language when processing mathematical questions, failing to fully grasp the meaning and requirements of the questions, resulting in poor accuracy and efficiency in answering.

[0068] To address this, the present invention provides a training method for a question-answering model. The method aims to improve the model's ability to understand and learn knowledge, answer questions, and predict the location and content of text in images through pre-training. The model is fine-tuned using training data during the training phase to better adapt it to question-answering tasks. It can extract the question text from any input question image and answer the question, thus improving answering efficiency while ensuring accuracy.

[0069] Figure 1 This is a flowchart illustrating the training method of the question-answering model provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0070] Step 110: Obtain pre-training data and training data. The pre-training data includes the first test question text and its corresponding first answer text, as well as the second test question image and the position and text content of each text line therein. The training data includes the third test question image, as well as the third test question text and the third answer text corresponding to the third test question image.

[0071] Specifically, before training the model, training data needs to be acquired. In this embodiment of the invention, to ensure the model's generalization performance and accuracy in answering tasks, the overall training process can be divided into two stages: a pre-training stage and a training stage. The pre-training stage improves the model's ability to understand the test text, learn the knowledge involved in the test, and answer questions. It also improves the model's ability to predict the text position and content in the input image. Based on this, downstream fine-tuning is performed. By fine-tuning the model parameters, it can be better adapted to downstream answering tasks, thereby improving the model's answering accuracy without affecting its generalization performance.

[0072] Based on this, in this embodiment of the invention, it is first necessary to obtain training samples for each stage, namely, sample data for the pre-training stage, i.e., pre-training data, and sample data for the training stage, i.e., training data. Since the pre-training stage requires learning in aspects such as answering questions, text position prediction, and text content prediction, the sample data for the pre-training stage must contain data that enables the model to learn in the above multiple aspects. For example, it may contain test text for answering ability training and its corresponding answer text, or test images for text position prediction and text content prediction tasks, as well as the position and text content of each text line therein.

[0073] That is, the pre-training data includes the first test question text and its corresponding first answer text used for answering ability training, and the second test question image used for text position prediction and text content prediction tasks, as well as the position of each text line and the text content of each text line in the second test question image. It is worth noting that the first test question text for answering ability training and the test question text in the second test question image used for text position prediction and text content prediction tasks can be the same or different, and can be selected according to the actual situation.

[0074] Correspondingly, the sample data during the training phase includes data that enables the model to better adapt to the question-answering task, allowing it to process the input image, extract the question text, and provide answers. For example, it may include the question image, the question text corresponding to the text extraction task, and the answer text corresponding to the question-answering task. That is, the training data includes the third question image, the third question text corresponding to the third question image, and the third answer text. Similarly, the third question image and the second question image here can be the same or different.

[0075] In the training samples at each stage, the test question images (the second test question image and the third test question image) can be obtained by acquiring images of existing paper test questions through image acquisition devices, or by searching, crawling, or directly downloading / capturing from specific test question websites, or by searching from test question databases. This embodiment of the invention does not make specific limitations in this regard.

[0076] The test text (the text content of each line in the first test text, the third test text, and the second test text) can be obtained by manual input, by extracting text from the corresponding test image, or by directly obtaining it from a test bank or a specific test website.

[0077] The answer texts (first answer text and third answer text) can be annotated by experienced experts, teachers, etc., or they can be obtained along with the corresponding test question text or test question image when obtaining the test question text or test question image. That is, the test questions and their corresponding standard answers can be obtained directly from the test question bank or specific website that carries the standard answers as the answer texts. Alternatively, the answers can be extracted from paper materials such as textbooks and teaching aids that contain test question answers. This embodiment of the invention does not make specific limitations in this regard.

[0078] The positions of each text line in the test question image can be manually labeled or detected by an object detection algorithm; this embodiment of the invention does not impose specific limitations on this. Correspondingly, the text content of each text line can also be determined by manual input or by text extraction and text recognition algorithms; this embodiment of the invention does not impose specific limitations on this.

[0079] Step 120: Train the initial model based on the pre-training data to obtain the pre-trained model;

[0080] Specifically, after obtaining the pre-training data in step 110, the model can be pre-trained based on the pre-training data to obtain the pre-trained model, i.e., the pre-trained model.

[0081] It is worth noting that before using pre-training data for model pre-training, the initial model needs to be determined first. Considering that traditional solutions can only answer questions with plain text, and the model's input and output do not involve images, it is impossible to process images, thus failing to effectively answer questions containing images. In this embodiment of the invention, in order to achieve automatic answering of questions containing images, such as matching questions and clock questions, an image encoder capable of image processing can be added when building the initial model. This image encoding is used to understand the semantic information contained therein, thereby enabling the answering of questions with images.

[0082] Based on this, in this embodiment of the invention, an image encoder can be added when constructing the initial model, and a language model, such as a large language model or a cognitive model, can be combined on this basis to construct the initial model before pre-training.

[0083] It is understandable that the purpose of using pre-training data to pre-train the model is to improve the model's ability to answer questions and predict the location and content of text in the input image. Therefore, the entire pre-training process can be regarded as pre-training the model for answering questions, as well as pre-training for text content prediction and text location prediction. The inputs and outputs of these two are not the same. Therefore, in this embodiment of the invention, the pre-training process can be segmented into two stages: the pre-training stage for answering questions and the pre-training stage for prediction.

[0084] The pre-training stage involves using question-answer text pairs to train the model, enabling it to respond to input question texts and output corresponding answer texts. By comparing the model's predicted answer texts with the answer labels, the model parameters are adjusted so that the output answer texts for input question texts are as close as possible to or consistent with the answer labels, thus giving the model excellent question-answering capabilities and resulting in a fully trained model.

[0085] The prediction pre-training stage uses a triplet of test image-text location-text content to train the model. This enables the model to predict the text location and recognize the text content of the input test image, and output the location and content of each text line. By comparing the predicted location and text content with the location label and content label, the model parameters are adjusted so that the output text line location and text content of the input test image can be as close as possible to the label, or consistent with the label. This gives the model the ability to predict text spatial location and recognize text content, and finally, the trained model is obtained.

[0086] It is worth noting that the training process in the two stages mentioned above can be performed simultaneously or separately, one after the other. For example, pre-training for answering questions can be performed first, followed by pre-training for prediction, or pre-training for prediction can be performed first, followed by pre-training for answering questions. This embodiment of the invention does not impose any specific limitations on this.

[0087] Specifically, this can be done by using the first question text and its corresponding first answer text to pre-train the initial model for answering questions, and by using the second question image, as well as the position and content of each text line in the second question image, to pre-train the initial model for prediction, so that the model has excellent answering ability, as well as the ability to predict text spatial position and recognize text content. Finally, the pre-trained model after the pre-training stage is completed can be obtained.

[0088] Step 130: Based on the pre-trained model, determine the pre-test question text and predicted answer text corresponding to the third test question image;

[0089] Specifically, after the pre-training in step 120, once the pre-trained model is obtained, downstream tasks can be performed based on the training data to determine the pre-test question text and predicted answer text corresponding to the training data.

[0090] Understandably, after pre-training is completed, the training phase can begin, whereby the pre-trained model can be trained using the training data to obtain the final answer model. In this process, the pre-trained model can first be used to process the training data to obtain the predicted output corresponding to the training data, namely the predicted test question text and predicted answer text corresponding to the third test question image in the training data.

[0091] Specifically, this can involve inputting the third question image into a pre-trained model, enabling the model to extract text and automatically answer questions from the input third question image. Specifically, this involves extracting structured information from the third question image to obtain structured question text, i.e., pre-test question text. Answering questions based on this text yields the corresponding answer text, i.e., predicted answer text.

[0092] Step 140: Based on the predicted test question text and predicted answer text corresponding to the third test question image, as well as the third test question text and the third answer text, fine-tune the pre-trained model to obtain the answer model.

[0093] Specifically, after obtaining the predicted test question text and predicted answer text corresponding to the third test question image predicted by the pre-trained model through the above process, the pre-trained model can be fine-tuned based on the predicted test question text, predicted answer text, and training data to obtain the answer model.

[0094] Understandably, after the above prediction process, once the pre-trained model's prediction output for the training data is obtained, the model's loss in the above prediction process can be measured using the model's prediction results and the labels in the training data. Based on the loss, the model parameters can be fine-tuned to finally obtain the trained answering model.

[0095] Specifically, we can first use the predicted test question text and predicted answer text corresponding to the third test question image, as well as the third test question text and third answer text corresponding to the third test question image in the training data, to determine the loss of the pre-trained model in the prediction task during the training phase. Then, based on this loss, we can iterate the parameters of the pre-trained model to obtain the final trained answer model.

[0096] The above process essentially measures the model's loss in the prediction task by comparing the difference between the model's output and the labels of the input images. This loss reflects the model's shortcomings in the prediction task, which can then be used to fine-tune the model's parameters. This fine-tuning aims to make the output question text and answer text corresponding to the input question image closer to the labels in the training data. Specifically, it aims to make the pre-trained model's output question text and predicted answer text corresponding to the input third question image as close as possible to the third question text and the third answer text. Through continuous prediction output and parameter fine-tuning, the model's prediction results can eventually approach the labels infinitely, or even perfectly match them. This final model is the answer model.

[0097] The question-answering model training method provided by this invention improves the initial model's ability to understand and learn knowledge, answer questions, and predict the location and content of text in images through pre-training. The initial model is then fine-tuned using training data during the training phase to better adapt it to question-answering tasks. This enables the model to answer questions containing images, overcoming the limitation of traditional methods that cannot handle such questions. The trained question-answering model effectively answers questions of any type, improving efficiency while ensuring accuracy. This benefits student learning efficiency and optimizes the user experience of related learning devices.

[0098] Based on the above embodiments, step 130 includes:

[0099] Based on the pre-trained model, determine the pre-test question text corresponding to the third test question image;

[0100] Based on the pre-trained model, the third question image and its corresponding pre-test question text, the predicted answer text corresponding to the third question image is determined.

[0101] Specifically, step 130, the process of determining the predicted test question text and predicted answer text corresponding to the third test question image based on the pre-trained model, may include:

[0102] First, the test text corresponding to the third test question image can be determined through a pre-trained model. That is, the test text predicted and output by the model is called the pre-test text corresponding to the third test question image. In other words, based on the third test question image, the pre-trained model is applied to predict the test text to obtain the pre-test text output by the model.

[0103] Specifically, the third test question image can be used as input and fed into a pre-trained model so that the pre-trained model can extract text from the input third test question image, that is, extract the structured information of the test question from the third test question image, so as to extract the structured test question information of the text, thereby obtaining the structured test question text, that is, the pre-test question text corresponding to the third test question image.

[0104] Next, based on this pre-test question text, the corresponding predicted answer text can be determined through a pre-trained model. That is, using the third question image and its corresponding pre-test question text as a benchmark, the prediction training model is applied to answer the question, thereby obtaining the predicted answer text corresponding to the third question image.

[0105] Specifically, the third question image and its corresponding predicted question text can be used as input to the pre-trained model, so that the pre-trained model can answer based on the input third question image and the corresponding structured question text, and output its answer text accordingly, thereby obtaining the predicted answer text corresponding to the third question image.

[0106] Based on the above embodiments, step 120 includes:

[0107] Based on the initial model, determine the predicted answer text corresponding to the first question text;

[0108] Based on the predicted answer text corresponding to the first test question text and the first answer text, the initial model is trained to obtain the first pre-trained model;

[0109] Based on the first pre-trained model, the predicted position of each text line in the second test question image and the predicted content of each text line are determined.

[0110] Based on the position and content of each text line, as well as the predicted position and predicted content, the first pre-trained model is trained to obtain the pre-trained model.

[0111] Specifically, step 120, the process of training the initial model based on the pre-training data to obtain the pre-trained model, may include:

[0112] First, the initial model can be used to determine the answer text corresponding to the first question text, that is, the answer text predicted by the initial model for the first question text, that is, the predicted answer text corresponding to the first question text. In other words, based on the first question text, the initial model is applied to generate the answer so that the initial model can answer the input first question text, thereby obtaining its predicted answer text.

[0113] Then, the predicted answer text and the first answer text corresponding to the first question image can be used to train the initial model to obtain the model that has been trained in this stage, namely the first pre-trained model. Specifically, the loss of the initial model in the question-answering pre-training stage is determined by the predicted answer text corresponding to the first question image and the first answer text. Based on this loss, the parameters are adjusted to obtain the first pre-trained model.

[0114] Specifically, this pre-training process uses the first question text and its corresponding first answer text as pre-training data. The first question text is used as input to the initial model so that the initial model can understand the input first question text, answer it, and output the corresponding answer text, i.e., the predicted answer text. Then, the loss of the initial model in the question-answering process can be determined using the first answer text and the predicted answer text. Based on this loss, the parameters are iterated to obtain the first pre-trained model.

[0115] It's worth noting that the aforementioned pre-training for answering questions can improve the model's question-answering ability. An essential step in the question-answering process is understanding the input question text and learning the relevant knowledge. Therefore, pre-training not only gives the model excellent question-answering ability but also enhances its understanding and learning capabilities. Understanding the input text primarily involves the language model in the initial model, while the image encoder processes images. Therefore, the question-answering pre-training here essentially targets the language model within the initial model. By pre-training the language model for answering questions, not only can its question-answering ability be significantly improved, but it can also acquire stronger comprehension capabilities, enabling it to fully interpret the semantics of the input question and accurately understand it, thus ensuring the accuracy of subsequent answers.

[0116] Then, the first pre-trained model can be pre-trained for prediction to improve its performance in text spatial location prediction and text content recognition, so as to obtain the final pre-trained model. That is, the second test question image in the pre-training data, as well as the position and text content of each text line in the second test question image, can be used to pre-train the first pre-trained model for prediction, so as to obtain the model that has been trained in this stage, i.e., the pre-trained model.

[0117] Specifically, the position of each text line in the second test question image can be determined first through the first pre-trained model. That is, based on the second test question image, the first pre-trained model is used to predict the text position to obtain the position of each text line in the second test question image, i.e., the predicted position. At the same time, the text content of each text line in the second test question image can be predicted through the first pre-trained model, thereby obtaining the text content predicted by the model, i.e., the predicted text content.

[0118] That is, for the input second test question image, the first pre-trained model can first predict the text position and recognize the text content, for example, by using an OCR engine to predict and recognize the text, thereby obtaining the predicted position (top left corner coordinates (x1, y1) and bottom right corner coordinates (x2, y2)) and the predicted content of each text line. Then, the text lines can be sorted according to the coordinates from top to bottom and from left to right, and the predicted position and predicted content are output in sequence. That is, for each input second test question image, the model outputs the predicted position and predicted content of the text lines in the order of coordinates.

[0119] Then, based on this, the position and content can be predicted, as well as the position and content of each text line in the second question image, to train the first pre-trained model and obtain the pre-trained model.

[0120] That is, based on the predicted location and content, as well as the location and content of each text line in the second question image, the loss of the initial model in the prediction pre-training stage is measured. The parameters are then adjusted based on this loss to obtain the pre-trained model. Specifically, this can be achieved by comparing the location and text content predicted by the first pre-trained model with the location and content of each text line in the second question image to determine the model's loss in spatial location prediction and text content recognition. The parameters of the first pre-trained model are then adjusted based on this loss so that the model with adjusted parameters can output predicted locations and content as closely as possible to the location and content of each text line for the input second question image. Finally, the model trained in this stage, i.e., the pre-trained model, is obtained.

[0121] Based on the above embodiments, in step 140, the pre-trained model is fine-tuned based on the predicted test question text and predicted answer text corresponding to the third test question image, as well as the third test question text and the third answer text, to obtain the answer model. This step further includes:

[0122] Obtain the answer image, along with the corresponding question text and answer result;

[0123] Based on the answer model, the predicted question text corresponding to the answer image is determined;

[0124] Based on the answer model, the answer image and its corresponding pre-test question text, the predicted answer result corresponding to the answer image is determined;

[0125] Based on the pre-test question text and predicted answer result corresponding to the answer image, as well as the test question text and answer result, the answer model is fine-tuned to obtain the review model.

[0126] Specifically, after obtaining the answer model through the above pre-training and training stages, considering that students usually face not only the need to obtain answers to unknown questions, but also the need to review the answers they have already answered, that is, the need to correct the answers, in this embodiment of the invention, after obtaining the answer model, further training can be carried out on this basis to obtain a review model with review capabilities.

[0127] Considering the current complex process of grading math problems, and the fact that it can only grade problems with plain text answers, the general process is as follows: (1) Obtain the student's answer image; (2) Obtain the question area and the answer area; (3) Perform OCR recognition on the question area and the answer area to obtain the question text and the answer text respectively; (4) Obtain the corresponding standard answer text from the question bank based on the question text; (5) Compare the answer text with the standard answer text to achieve grading. This process first requires a large question bank to store the questions and the corresponding standard answers. At the same time, a matching algorithm needs to be designed to find the question text with the same text content from the question bank based on the recognized question text. If the question is mismatched, the wrong standard answer will be obtained, affecting the accuracy of grading. Correspondingly, if the corresponding question text cannot be found from the question bank, the standard answer cannot be obtained, and the subsequent grading process cannot be executed, thus failing to achieve grading of the answer questions.

[0128] Therefore, in this embodiment of the invention, it is proposed that the answering model can be further fine-tuned to make it suitable for review tasks, thus obtaining a review model. Based on the training concept of answering questions first and then reviewing, this embodiment not only enables the training of both answering and reviewing models under the same architecture, but also allows the trained review model to eliminate dependence on a question bank during review. Furthermore, it can review questions of any type and directly perform text recognition on the input answer image without relying on an OCR engine, simplifying the review process, improving review efficiency, and avoiding the overhead of question bank storage.

[0129] Specifically, after obtaining the answer model and before training it to obtain the review model, training data for this training stage needs to be acquired, namely, answer images, their corresponding question texts, and answer results. Here, the answer images can be images directly collected from students' answers to the questions, or images downloaded / crawled from specific websites, question banks, etc.; the corresponding question texts can be text recognition performed on the collected answer images, or they can be obtained directly from specific websites, question banks, etc., when crawling question images, or they can be manually annotated. The answer results can be annotated by experienced experts, teachers, etc., or they can be obtained along with the question images from websites, question banks, etc., containing questions and answer results indicating whether the answers are correct or not. This embodiment of the invention does not specifically limit this.

[0130] After obtaining the answer image, the answer model can be used to determine the corresponding question text, that is, the text content of the question in the answer image predicted by the answer model, i.e., the pre-test question text. Specifically, the answer image can be input into the answer model so that the answer model can extract text from the input answer image, that is, extract the structured information of the question from the answer image, so as to obtain the structured question text, i.e., the pre-test question text corresponding to the answer image.

[0131] Then, based on this pre-test question text, the corresponding predicted answer result can be determined through the answer model. That is, using the answer image and its corresponding pre-test question text as a benchmark, the answer model reviews the student's answers in the answer image to obtain the predicted answer result. Specifically, the answer image and its corresponding pre-test question text are both input into the answer model, allowing the model to review the answers based on the input answer image and the corresponding structured test text, and output its review result, thus obtaining the predicted answer result corresponding to the answer image. The predicted answer result here represents the judgment result of whether the student's answer is correct, such as right or wrong.

[0132] Subsequently, based on the pre-test question text and predicted answer result corresponding to the answer image output by the above answer model, as well as the test question text and answer result corresponding to the answer image in the training data, the loss of the model in the above review stage can be measured, so as to fine-tune the model parameters according to the loss, and finally obtain the trained review model.

[0133] The above process essentially measures the model's loss in the answer review task by comparing the model's output with the label of the input answer image. This loss reflects the model's shortcomings in the answer review task, and the model's parameters can then be adjusted accordingly to ensure that the model's output of the predicted test text and the predicted answer result corresponding to the input answer image is as close as possible to the test text and the answer result corresponding to the answer image. Through continuous review output and parameter fine-tuning, the model's predicted output result can eventually approach the label infinitely, or be completely consistent with the label. At this point, the answer model is the final model required, namely the review model.

[0134] In this embodiment of the invention, a review model is obtained by training on the answer model. This not only realizes the two tasks of answering and reviewing under the same architecture, but also simplifies the process. It eliminates the need for an additional OCR system for text recognition and avoids the high cost of storing a question bank, thus avoiding the various problems caused by question bank matching in traditional solutions.

[0135] Based on the above embodiments, the initial model was built on the basis of the image encoder and the cognitive model.

[0136] Currently, most automated question answering systems use relatively small language models to process input text and output corresponding answers. However, small language models often fall short when dealing with complex problems. Specifically, they struggle to fully understand complex and diverse mathematical questions, resulting in poor accuracy. Furthermore, when question images are added, their ability to correlate and interact with image and text features is also weak, leading to poor final answering results.

[0137] Therefore, in this embodiment of the invention, when constructing the initial model, a large language model with strong comprehension ability and capable of fully understanding complex and diverse test questions can be selected, such as a cognitive model. The initial model is constructed by the cognitive model and the image encoder.

[0138] Figure 2 This is a general framework diagram of the training method for the question-answering model provided by the present invention, as shown below. Figure 2 As shown, the image encoder can encode the input test question image into visual features, i.e., image features, which are then input into the cognitive model for feature interaction. Through attention interaction with text features in the cognitive model, corresponding outputs can be obtained, such as predicted test question text, predicted answer text, and predicted answer results.

[0139] Among them, the text features can be features obtained by text encoding the prompt text. The prompt text is used to prompt the initial model to make corresponding outputs based on the input. The prompt text can be "Please extract the structured question text based on the given image" or "[Structured question text] Please answer the question based on the image information and the extracted question information", etc. The embodiments of the present invention do not make specific limitations on this.

[0140] Here, the image encoder can be a feature extraction model based on a convolutional neural network, a feature extraction model based on a transformer, or a Swing Transformer; this embodiment of the invention does not specifically limit this. Preferably, this embodiment uses a Swing Transformer as the image encoder.

[0141] It is understood that the end-to-end model adopted in this embodiment of the invention is based on an encoder-decoder architecture, where the image encoder is the encoding end and the cognitive model is the decoding end, acting as the decoder. Here, the cognitive model is a transformer model trained on an autoregressive basis, with a large number of parameters and has undergone large-scale pre-training on a large corpus of natural language. Therefore, it contains a large amount of factual knowledge and common sense, and can generate corresponding responses based on prompt text. In addition, it also has context learning capabilities, and can learn to understand such questions from a given set of examples, which is something that traditional language models do not possess.

[0142] In this embodiment of the invention, the cognitive model is designed with targeted input and output, enabling the model to output the question text and answer text corresponding to the input question image. After training through answering questions, the model can better understand the question text and learn to answer. In addition, the feature interaction based on encoder-decoder can also enable the model to understand the text, graphics, etc. in the input question image, and has visual perception ability. Therefore, it can understand mathematical problems more accurately, has strong thinking ability, and can answer questions quickly and accurately.

[0143] This invention also provides a method for answering questions. Figure 3 This is a flowchart illustrating the answering method provided by the present invention, as shown below. Figure 3 As shown, the method includes:

[0144] Step 310: Determine the image of the question to be answered;

[0145] Step 320: Based on the answer model, determine the answer text corresponding to the question image; the answer model is determined based on the training method of the answer model as described in any of the above items.

[0146] Specifically, before answering the questions, it is necessary to first determine the questions that need to be answered automatically, i.e. the questions to be answered. This can be done by acquiring the image of the questions to be answered. The image of the questions can be obtained by acquiring the image of existing paper questions through an image acquisition device, or by searching, crawling, or directly downloading / capturing from a specific question website, or by searching from a question bank. This embodiment of the invention does not make specific limitations in this regard.

[0147] Here, the image containing the questions to be answered can contain one or more questions. If it contains multiple questions, the entire image can be cropped into multiple blocks, each containing one question. This allows the answering model to process each question and output its corresponding answer text when applying it later. It's worth noting that the image can be either one or multiple. If there are multiple images, the answer can be performed on each image to obtain the corresponding answer text for each question.

[0148] Furthermore, after determining the question image to be answered, the questions contained therein can be answered to obtain the corresponding answer text. That is, a pre-trained answering model can be applied to answer the questions and obtain the answer text corresponding to the question image. Specifically, the question image can be input into the answering model so that the answering model can understand the input question image and answer it, thereby obtaining the predicted output answer text, that is, the answer text corresponding to the question image.

[0149] Here, the answering model can first extract text from the input question image to extract the structured information of the question, thereby obtaining the structured question text. Then, it can answer the question based on the structured question text and the question image, and finally obtain the corresponding answer text.

[0150] The answering model can be trained using the following steps: First, acquire pre-training data and training data. The pre-training data includes the first question text and its corresponding first answer text, as well as the second question image and the position and content of each text line within it. The training data includes the third question image, and the third question text and third answer text corresponding to the third question image. Second, train the initial model based on the pre-training data to obtain the pre-trained model. Third, determine the predicted question text and predicted answer text corresponding to the third question image based on the pre-trained model. Fourth, fine-tune the pre-trained model based on the predicted question text and predicted answer text corresponding to the third question image, as well as the third question text and third answer text, to obtain the answering model.

[0151] The question-answering method provided by this invention applies a question-answering model trained using a question-answering model training method to answer questions on test question images, obtaining the corresponding answer text. This overcomes the shortcomings of traditional solutions that cannot answer questions containing images, enabling automatic answering of questions of any type without relying on an additional OCR system. This simplifies the question-answering process while ensuring accuracy and efficiency, thus contributing to the optimization of students' learning efficiency and the user experience of related learning devices.

[0152] Based on the above embodiments, step 320 includes:

[0153] Based on the answer model, determine the features of the question image corresponding to the question image;

[0154] Identify the test question prompt text and determine the text features based on the test question prompt text;

[0155] Based on the answer model, as well as the image and text features of the test questions, the answer text corresponding to the test question image is determined.

[0156] Specifically, step 320, the process of determining the answer text corresponding to the question image based on the answer model, may include:

[0157] First, the image of the question to be answered can be input into the answering model. The answering model can then perform visual encoding on the input question image to encode it into visual features, thereby obtaining the question image features corresponding to the question image.

[0158] Simultaneously, the prompt text can be determined so that the model can extract structured question information from the question image based on the prompt text, thereby obtaining the question text. Then, the model can answer the question based on the question text to obtain the answer text, that is, the answer text corresponding to the question image.

[0159] Specifically, the question prompt text can be determined first. This prompt text is used to guide the answering model to extract the question text from the input question image. It can be "Please extract the structured question text based on the given image", or "Please extract the structured question text based on the provided question image", or other forms. This embodiment of the invention does not specifically limit this.

[0160] Then, text features can be determined based on the question prompt text. Specifically, the question prompt text can be processed by a tokenizer to obtain the embedded representation of the question prompt text, which can be used as a text feature to interact with the question image features. That is, cross-modal interaction of features is carried out in the answer model to obtain the question text and answer text output by the answer model.

[0161] In other words, after obtaining the text features, an answering model can be applied based on these text features and the visually encoded question image features to automatically answer questions and obtain the answer text corresponding to the question image. Specifically, this can be achieved by first performing feature interaction in the answering model, through cross-modal interaction between text features and question image features, so that the model can fully understand the question prompt text and question image, and output structured question text. Based on this, the answering model can be applied to answer questions by using the extracted question text, thereby obtaining the answer text corresponding to the question image.

[0162] Based on the above embodiments, and based on the answer model, as well as the image features and text features of the test questions, the answer text corresponding to the test question image is determined, including:

[0163] Based on the question-answering model, attention interaction is performed on the question image features and text features to obtain the question text corresponding to the question image;

[0164] Based on the test question text, determine the answer prompt text, and based on the answer prompt text, determine the characteristics of the answer text;

[0165] Based on the question-answering model, attention interaction is performed on the features of the question image and the features of the answer text to obtain the answer text corresponding to the question image.

[0166] Specifically, the process of determining the answer text corresponding to the question image based on the answer model, as well as the image and text features of the question, includes the following steps:

[0167] First, after obtaining the image features and text features of the test questions, feature interaction can be performed in the answering model. This allows the model to output the test text in the test question image based on the input test question image and text features. In other words, the answering model can perform attention interaction on the test question image features and text features to obtain the test question text corresponding to the test question image.

[0168] Here, the cognitive model in the trained question-answering model essentially acts as a decoder. It interacts with the text features determined by the question prompts, based on the image features encoded by the image encoder. This cross-modal feature interaction enhances the model's understanding of the input information, leading to better output. The feature interaction process is actually achieved within the cognitive model through an attention mechanism that fuses the question image features and text features. This cognitive model is essentially a language model with strong comprehension and rich knowledge, structurally similar to BERT (Bidirectional Encoder Representations from Transformers). It contains multiple attention layers that interact with the input question image features and text features, ultimately producing the model's output question text.

[0169] After obtaining the question text, an answer model can be applied to answer the question and determine the answer text. Here, the answer prompt text can be determined first by extracting the question text, that is, the extracted question text is used as part of the answer prompt text. The answer prompt text is used to prompt the answer model to answer the question based on the input information. It can be "[Structured question text] Please answer the question based on the image information and the extracted question information", or "[Extracted question text] Please answer the question by combining the question image and the extracted question text", or other forms. This embodiment of the invention does not specifically limit this.

[0170] Then, the answer text features can be determined based on the answer prompt text. Specifically, the answer prompt text can be processed by a tokenizer to obtain the embedded representation of the answer prompt text, which can be used as the answer text feature to interact with the test question image features. Through cross-modal feature interaction in the answer model, the answer text corresponding to the test question image output by the answer model can be obtained.

[0171] That is, after obtaining the answer text features, the answer model can be applied to perform attention interaction between the question image features and the answer text features. Through cross-modal interaction of multiple attention layers in the answer model, the model can fully understand the question image and the answer prompt text, thereby answering the question and outputting the corresponding answer text.

[0172] It is worth noting that for the text features and answer text features of the cognitive model in the input question-answering model, since the prompt text is embedded in the word segmenter, the information contained therein is usually not rich enough. Therefore, after inputting them into the question-answering model, the model can first process the input text features and answer text features to enrich the semantic information contained therein, so that the subsequent model can understand them during the interaction process and thus output more accurately. After that, the processed text features or answer text features can be interacted with the question image features through attention to output the question text or answer text corresponding to the question image.

[0173] The following example illustrates the answering process in detail:

[0174] Figure 4 This is a flowchart illustrating the question-answering process provided by the present invention, such as... Figure 4 As shown, when applying the answering model to answer questions, the image of the question to be answered can be determined first, and the corresponding question prompt text can be determined, such as "Please extract the structured question text based on the given image"; then, the text features can be determined based on the question prompt text. At the same time, the answering model can perform visual encoding on the question image to obtain the question image features; after that, the answering model can perform attention interaction on the question image features and text features, and output the question text, such as "Which of the following formulas can be used to represent the following meaning? () A. 3+2=5 B. 12-3=10 C. 13-3-2=8".

[0175] Then, the answer prompt text can be determined from the question text, such as "[Which of the following equations can represent the following meaning? () A. 3+2=5 B. 12-3=10 C. 13-3-2=8] Please answer the question based on the image information and the extracted question information." The answer text features can then be determined based on the answer prompt text, and then input into the answer model. The answer model can then perform attention interaction on the question image features and answer text features, and output the answer text, such as " <box> xlylx2y2< / box> "C", which can then be mapped back to the input question image to obtain the final output.

[0176] Based on the above embodiments, the method further includes:

[0177] Identify the answer images to be reviewed;

[0178] Based on the review model, determine the answer image features corresponding to the answer image;

[0179] Identify the test question prompt text and determine the text features based on the test question prompt text;

[0180] Based on the review model, as well as the features of the answer image and the text, the answer result corresponding to the answer image is determined.

[0181] Specifically, in this embodiment of the invention, in addition to the above-mentioned answering method, a review model obtained by further training on the answering model can also be applied for review, that is, the student's answer image can also be reviewed to obtain the answer result, that is, whether the student's answer is correct or not.

[0182] Based on this, in this embodiment of the invention, before reviewing, the student's answer image that needs to be reviewed can be determined first, that is, the answer image to be reviewed; here, the answer image can be an image of the student answering the question directly, or it can be an image of the student answering the question crawled from a specific website, question bank, etc. This embodiment of the invention does not limit this.

[0183] After obtaining the answer images to be reviewed, the students' answers can be reviewed to obtain the corresponding review results, i.e., the answer results. Specifically, a pre-trained review model can be applied to review the answers and obtain the answer results corresponding to the answer images. Here, the answer images can be input into the review model so that the model can understand and review the input answer images, thereby predicting the output review results.

[0184] Specifically, the answer image to be reviewed is first input into the review model. The review model can then perform visual encoding on the input answer image to encode it into visual features, thereby obtaining the answer image features corresponding to the answer image.

[0185] Simultaneously, the prompt text can be determined so that the model can extract structured test question information from the answer image based on the prompt text, thereby obtaining the test question text. The model can then review the test question text and the answer image to obtain the review result, which is the answer result corresponding to the answer image.

[0186] That is, the question prompt text can be determined first. This question prompt text is used to prompt the review model to extract the question text from the input answer image. It can be "Please extract the structured question text based on the given image", or "Please extract the structured question text based on the provided answer image", or other forms. This embodiment of the invention does not make specific limitations on this.

[0187] Then, text features can be determined based on the question prompt text. Specifically, the question prompt text can be processed by a tokenizer to obtain the embedded representation of the question prompt text, which can be used as a text feature to interact with the answer image features. That is, cross-modal interaction of features is carried out in the review model to obtain the question text and answer result output by the review model.

[0188] In other words, after obtaining the text features, an automatic review model can be applied based on these text features and the answer image features to obtain the answer result corresponding to the answer image. Specifically, feature interaction can first be performed in the review model. Through cross-modal interaction between text features and answer image features, the model can understand the question prompt text and answer image, and output structured question text. Based on this, the review model is applied to review the text, and combined with the answer image, the answer result corresponding to the answer image is obtained.

[0189] The method provided by this invention applies a review model trained on the answer model to review answers, realizing automatic review of any question type, simplifying the review process, eliminating the need for an additional OCR system for text recognition, and avoiding the high cost of storing a question bank, thus avoiding various problems caused by question bank matching in traditional review schemes.

[0190] Based on the above embodiments, and based on the review model, as well as the answer image features and text features, the answer result corresponding to the answer image is determined, including:

[0191] Based on the review model, attention interaction is performed on the features of the answer image and the text features to obtain the test text corresponding to the answer image;

[0192] Based on the test question text corresponding to the answer image, the review prompt text is determined, and the review text features are determined based on the review prompt text;

[0193] Based on the review model, attention interaction is performed on the features of the answer image and the features of the review text to obtain the answer result corresponding to the answer image.

[0194] Specifically, the process of determining the answer result corresponding to the answer image based on the review model, as well as the answer image features and text features, includes the following steps:

[0195] First, after obtaining the answer image features and text features, feature interaction can be performed in the review model. This allows the model to output the test text in the answer image based on the input answer image and text features. In other words, the review model can be used to perform attention interaction on the answer image features and text features to obtain the test text corresponding to the answer image.

[0196] Here, the cognitive model in the trained review model essentially acts as a decoder. It interacts with the text features determined by the question image encoded by the image encoder and the question prompt text. Through cross-modal feature interaction, the model's understanding of the input information is improved, leading to better output. The feature interaction process is actually achieved within the cognitive model through an attention mechanism that fuses the answer image features and text features. This cognitive model is essentially a language model with strong comprehension capabilities and rich knowledge, structurally similar to BERT. It contains multiple attention layers that interact with the input answer image features and text features, ultimately producing the output question text.

[0197] After obtaining the test question text, a review model can be applied to answer the questions and determine the answer results. Here, the review prompt text can be determined first through the extracted test question text. That is, the extracted test question text is used as part of the review prompt text. This review prompt text is used to prompt the review model to review the answer based on the input information. It can be "[Structured Question Text] Please correct according to the image information and the extracted question information", or "[Extracted Test Question Text] Please review in combination with the answer image and the extracted test question text", or other forms. This embodiment of the invention does not specifically limit this.

[0198] Then, the review text features can be determined based on the review prompt text. Specifically, the review prompt text can be processed by a tokenizer to obtain the embedded representation of the review prompt text, which can be used as the review text feature to interact with the answer image features. Through cross-modal feature interaction in the review model, the answer result corresponding to the answer image output by the review model can be obtained.

[0199] That is, after obtaining the features of the review text, the review model can be applied to perform attention interaction between the features of the answer image and the features of the review text. Through cross-modal interaction of multiple attention layers in the review model, the model can fully understand the answer image and the review prompt text, thereby performing answer review and outputting the corresponding review result.

[0200] It is worth noting that for the text features of the cognitive model and the review text features input into the review model, since the prompt text is embedded in the word segmenter, the information contained therein is usually not rich enough. Therefore, after inputting them into the review model, the review model can first process the input text features and review text features to enrich the semantic information contained therein, so that the subsequent model can understand them during the interaction process and thus output more accurately. After that, the processed text features or review text features can be interacted with the answer image features through attention to output the test question text or answer result corresponding to the answer image.

[0201] The following example illustrates the review process in detail:

[0202] Figure 5 This is a flowchart illustrating the review process provided by the present invention, such as... Figure 5 As shown, when applying the review model for review, the first step is to determine the answer image to be reviewed and the corresponding question prompt text, such as "Please extract the structured question text based on the given image." Next, the text features can be determined based on the question prompt text. Simultaneously, the review model can visually encode the answer image to obtain answer image features. Afterward, the review model can perform attention interaction between the answer image features and text features, and output structured question text, such as "Draw the hour hand and branches on the clock face. (3 points)" <figure> 8:30 <figure> 16:00 <figure>22:00.

[0203] Then, the review prompts can be determined from the question text, such as "[Draw the hour hand and branches on the clock face. (3 points)]" <figure> 8:30 <figure> 16:00 <figure>[22:00] Please correct the answers based on the image information and the extracted question information. You can also determine the features of the review text based on the review prompts, and then input it into the review model. The review model can then perform attention interaction between the answer image features and the review text features, and output the answer result, such as... <box> xlylx2y2x3y3x4y4< / box> "yesbox>x5y5x6y6yes", which can then be mapped back to the input answer image to obtain the final output.

[0204] The training apparatus for the answer model provided by the present invention will be described below. The training apparatus for the answer model described below can be referred to in correspondence with the training method for the answer model described above.

[0205] Figure 6 This is a schematic diagram of the structure of the training device for the question-answering model provided by the present invention, as shown below. Figure 6 As shown, the device includes:

[0206] The acquisition unit 610 is used to acquire pre-training data and training data. The pre-training data includes a first test question text and its corresponding first answer text, as well as a second test question image and the position and text content of each text line therein. The training data includes a third test question image, as well as the third test question text and the third answer text corresponding to the third test question image.

[0207] The pre-training unit 620 is used to train the initial model based on the pre-training data to obtain a pre-trained model;

[0208] The prediction unit 630 is used to determine the predicted test question text and the predicted answer text corresponding to the third test question image based on the pre-trained model.

[0209] Training unit 640 is used to fine-tune the pre-trained model based on the predicted test question text and predicted answer text corresponding to the third test question image, as well as the third test question text and the third answer text, to obtain the answer model.

[0210] The training device for the answering model provided by this invention improves the initial model's ability to understand and learn knowledge, answer questions, and predict the location and content of text in images through pre-training. It fine-tunes the initial model using training data during the training phase to better adapt it to answering tasks, enabling it to answer questions containing images. This overcomes the limitation of traditional answering schemes that cannot answer questions containing images. The trained answering model achieves effective answers to questions of any type, improving answering efficiency while ensuring accuracy. This benefits student learning efficiency and optimizes the user experience of related learning devices.

[0211] Based on the above embodiments, the prediction unit 630 is used for:

[0212] Based on the pre-trained model, the pre-test question text corresponding to the third test question image is determined;

[0213] Based on the pre-trained model, the third test question image and its corresponding pre-test question text, the predicted answer text corresponding to the third test question image is determined.

[0214] Based on the above embodiments, the pre-training unit 620 is used for:

[0215] Based on the initial model, the predicted answer text corresponding to the first question text is determined;

[0216] Based on the predicted answer text corresponding to the first test question text and the first answer text, the initial model is trained to obtain the first pre-trained model;

[0217] Based on the first pre-trained model, the predicted position of each text line in the second test question image and the predicted content of each text line are determined.

[0218] Based on the position and content of each text line, as well as the predicted position and the predicted content, the first pre-trained model is trained to obtain the pre-trained model.

[0219] Based on the above embodiments, the training unit 640 is further configured to:

[0220] Obtain the answer image, as well as the corresponding question text and answer result;

[0221] Based on the answering model, the pre-test question text corresponding to the answering image is determined;

[0222] Based on the answering model, the answering image and its corresponding predicted question text, the predicted answering result corresponding to the answering image is determined;

[0223] Based on the predicted test question text and predicted answer result corresponding to the answer image, as well as the test question text and the answer result, the answer model is fine-tuned to obtain the review model.

[0224] Based on the above embodiments, the initial model is constructed on the basis of the image encoder and the cognitive model.

[0225] The answering device provided by the present invention is described below. The answering device described below and the answering method described above can be referred to in correspondence.

[0226] Figure 7 This is a schematic diagram of the answering device provided by the present invention, as shown below. Figure 7 As shown, the device includes:

[0227] Unit 710 is used to determine the image of the test question to be answered;

[0228] Answering unit 720 is used to determine the answer text corresponding to the question image based on the answering model; the answering model is determined based on the training method of the answering model as described above.

[0229] The answering device provided by this invention uses an answering model trained by a training method to answer questions on test question images and obtain the answer text corresponding to the test question image. This overcomes the shortcomings of traditional solutions that cannot answer questions containing images, realizes automatic answering of questions of any type, and does not rely on an additional OCR system, simplifying the answering process while ensuring answering accuracy and efficiency. This provides benefits to students' learning efficiency and optimizes the user experience of related learning devices.

[0230] Based on the above embodiments, the answering unit 720 is used for:

[0231] Based on the answering model, determine the question image features corresponding to the question image;

[0232] Determine the test question prompt text, and determine the text features based on the test question prompt text;

[0233] Based on the answer model, the question image features, and the text features, the answer text corresponding to the question image is determined.

[0234] Based on the above embodiments, the answering unit 720 is used for:

[0235] Based on the answering model, attention interaction is performed on the question image features and the text features to obtain the question text corresponding to the question image;

[0236] Based on the test question text, determine the answer prompt text, and based on the answer prompt text, determine the answer text features;

[0237] Based on the answering model, attention interaction is performed on the question image features and the answer text features to obtain the answer text corresponding to the question image.

[0238] Based on the above embodiments, the device further includes a review unit, used for:

[0239] Identify the answer images to be reviewed;

[0240] Based on the review model, the answer image features corresponding to the answer image are determined;

[0241] Determine the test question prompt text, and determine the text features based on the test question prompt text;

[0242] Based on the review model, the answer image features, and the text features, the answer result corresponding to the answer image is determined.

[0243] Based on the above embodiments, the review unit is used for:

[0244] Based on the review model, attention interaction is performed on the answer image features and the text features to obtain the test text corresponding to the answer image;

[0245] Based on the question text corresponding to the answer image, the review prompt text is determined, and the review text features are determined based on the review prompt text;

[0246] Based on the review model, attention interaction is performed on the features of the answer image and the features of the review text to obtain the answer result corresponding to the answer image.

[0247] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logic instructions in the memory 830 to execute a training method or a question-answering method for the answer model. The training method includes: acquiring pre-training data and training data, wherein the pre-training data includes a first question text and its corresponding first answer text, and a second question image and the position and content of each text line therein; the training data includes a third question image, and the third question text and third answer text corresponding to the third question image; training an initial model based on the pre-training data to obtain a pre-trained model; determining the predicted question text and predicted answer text corresponding to the third question image based on the pre-trained model; and fine-tuning the pre-trained model based on the predicted question text and predicted answer text corresponding to the third question image, as well as the third question text and the third answer text, to obtain the answer model. The question-answering method includes: determining a question image to be answered; determining the answer text corresponding to the question image based on the answer model; wherein the answer model is determined based on the training method for the answer model as described above.

[0248] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0249] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the training method or answering method of the answering model provided by the above methods, wherein the training method of the answering model comprises: acquiring pre-training data and training data, the pre-training data comprising a first test question text and its corresponding first answer text, and a second test question image and the position and text content of each text line therein, the training data comprising a third test question image, and a third test question text and a third answer text corresponding to the third test question image; training an initial model based on the pre-training data to obtain a pre-trained model; determining the predicted test question text and predicted answer text corresponding to the third test question image based on the pre-trained model; and fine-tuning the pre-trained model based on the predicted test question text and predicted answer text corresponding to the third test question image, as well as the third test question text and the third answer text, to obtain an answering model. The answering method includes: determining the image of the question to be answered; determining the answer text corresponding to the question image based on the answering model; wherein the answering model is determined based on the training method of the answering model as described above.

[0250] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a training method or answering method for the answering model provided by the above methods. The training method for the answering model includes: acquiring pre-training data and training data, wherein the pre-training data includes a first test question text and its corresponding first answer text, and a second test question image and the position and text content of each text line therein; the training data includes a third test question image, and a third test question text and a third answer text corresponding to the third test question image; training an initial model based on the pre-training data to obtain a pre-trained model; determining the predicted test question text and predicted answer text corresponding to the third test question image based on the pre-trained model; and fine-tuning the pre-trained model based on the predicted test question text and predicted answer text corresponding to the third test question image, as well as the third test question text and the third answer text, to obtain an answering model. The answering method includes: determining the image of the question to be answered; determining the answer text corresponding to the question image based on the answering model; wherein the answering model is determined based on the training method of the answering model as described above.

[0251] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0252] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0253] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / figure> < / figure> < / figure> < / figure> < / figure> < / figure>

Claims

1. A training method for a question-answering model, characterized in that, include: Acquire pre-training data and training data. The pre-training data includes a first test question text and its corresponding first answer text, as well as a second test question image and the position and text content of each text line therein. The training data includes a third test question image, as well as the third test question text and the third answer text corresponding to the third test question image. Based on the pre-training data, the initial model is trained to obtain the pre-trained model; Based on the pre-trained model, the predicted test question text and predicted answer text corresponding to the third test question image are determined; Based on the predicted test question text and predicted answer text corresponding to the third test question image, as well as the third test question text and the third answer text, the pre-trained model is fine-tuned to obtain the answer model; The step of determining the predicted test question text and predicted answer text corresponding to the third test question image based on the pre-trained model includes: Based on the pre-trained model, the pre-test question text corresponding to the third test question image is determined; Based on the pre-trained model, the third test question image and its corresponding pre-test question text, the predicted answer text corresponding to the third test question image is determined.

2. The training method for the answer model according to claim 1, characterized in that, The step of training the initial model based on the pre-trained data to obtain the pre-trained model includes: Based on the initial model, the predicted answer text corresponding to the first question text is determined; Based on the predicted answer text corresponding to the first test question text and the first answer text, the initial model is trained to obtain the first pre-trained model; Based on the first pre-trained model, the predicted position of each text line in the second test question image and the predicted content of each text line are determined. Based on the position and content of each text line, as well as the predicted position and the predicted content, the first pre-trained model is trained to obtain the pre-trained model.

3. The training method for the answer model according to claim 1 or 2, characterized in that, The pre-trained model is fine-tuned based on the predicted test question text and predicted answer text corresponding to the third test question image, as well as the third test question text and the third answer text, to obtain the answer model. This process further includes: Obtain the answer image, as well as the corresponding question text and answer result; Based on the answering model, the pre-test question text corresponding to the answering image is determined; Based on the answering model, the answering image and its corresponding predicted question text, the predicted answering result corresponding to the answering image is determined; Based on the predicted test question text and predicted answer result corresponding to the answer image, as well as the test question text and the answer result, the answer model is fine-tuned to obtain the review model.

4. The training method for the answer model according to claim 1 or 2, characterized in that, The initial model was built upon the image encoder and cognitive model.

5. A method for answering questions, characterized in that, include: Identify the image of the question to be answered; Based on the question-answering model, the answer text corresponding to the question image is determined; The answering model is determined based on the training method of the answering model as described in any one of claims 1 to 4.

6. The answering method according to claim 5, characterized in that, The step of determining the answer text corresponding to the question image based on the question-answering model includes: Based on the answering model, determine the question image features corresponding to the question image; Determine the test question prompt text, and determine the text features based on the test question prompt text; Based on the answer model, the question image features, and the text features, the answer text corresponding to the question image is determined.

7. The answering method according to claim 6, characterized in that, The step of determining the answer text corresponding to the question image based on the answer model, the question image features, and the text features includes: Based on the answering model, attention interaction is performed on the question image features and the text features to obtain the question text corresponding to the question image; Based on the test question text, determine the answer prompt text, and based on the answer prompt text, determine the answer text features; Based on the answering model, attention interaction is performed on the question image features and the answer text features to obtain the answer text corresponding to the question image.

8. The answering method according to claim 5, characterized in that, Also includes: Identify the answer images to be reviewed; Based on the review model, the answer image features corresponding to the answer image are determined; Determine the test question prompt text, and determine the text features based on the test question prompt text; Based on the review model, the answer image features, and the text features, the answer result corresponding to the answer image is determined; The review model is obtained by fine-tuning the answer model.

9. The answering method according to claim 8, characterized in that, The step of determining the answer result corresponding to the answer image based on the review model, the answer image features, and the text features includes: Based on the review model, attention interaction is performed on the answer image features and the text features to obtain the test text corresponding to the answer image; Based on the question text corresponding to the answer image, the review prompt text is determined, and the review text features are determined based on the review prompt text; Based on the review model, attention interaction is performed on the features of the answer image and the features of the review text to obtain the answer result corresponding to the answer image.

10. A training device for a question-answering model, characterized in that, include: The acquisition unit is used to acquire pre-training data and training data. The pre-training data includes a first test question text and its corresponding first answer text, as well as a second test question image and the position and text content of each text line therein. The training data includes a third test question image, as well as the third test question text and the third answer text corresponding to the third test question image. A pre-training unit is used to train an initial model based on the pre-training data to obtain a pre-trained model; The prediction unit is used to determine the predicted test question text and predicted answer text corresponding to the third test question image based on the pre-trained model. The training unit is used to fine-tune the pre-trained model based on the predicted test question text and predicted answer text corresponding to the third test question image, as well as the third test question text and the third answer text, to obtain the answer model; The step of determining the predicted test question text and predicted answer text corresponding to the third test question image based on the pre-trained model includes: Based on the pre-trained model, the pre-test question text corresponding to the third test question image is determined; Based on the pre-trained model, the third test question image and its corresponding pre-test question text, the predicted answer text corresponding to the third test question image is determined.

11. A question-answering device, characterized in that, include: The unit is used to determine the image of the question to be answered; The answering unit is used to determine the answer text corresponding to the question image based on the answering model; The answering model is determined based on the training method of the answering model as described in any one of claims 1 to 4.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the training method for the answering model as described in any one of claims 1 to 4, or the answering method as described in any one of claims 5 to 9.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the answering model as described in any one of claims 1 to 4, or the answering method as described in any one of claims 5 to 9.

Images