Answering model training method and device and answering method and device
By building the initial model of the visual encoder and language model, and using pre-training data and training data to pre-train and fine-tune the model, the problem of poor graphics information processing in the answers to geometry questions is solved, and efficient and accurate answers to the included graphics questions are achieved.
Patent Information
- Application Number
- CN202411637370.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-15
- Publication Date
- 2025-04-15
AI Technical Summary
When using large language models to solve geometric questions, the prior art is difficult to accurately understand and process graphic information, resulting in unsatisfactory answers.
By building an initial model containing a visual encoder and a language model, pre-training the model with pre-training data, the model's understanding of the graphics area and text information extraction ability, and fine-tuning the pre-trained model during the training stage to adapt to the question-answer task.
It realizes accurate answers to questions containing graphics, overcomes the shortcomings of inability to accurately answer questions containing graphics in traditional solutions, improves the efficiency and accuracy of answering questions, and is suitable for the optimization of students' learning and learning equipment.
Smart Images
Figure CN119832563A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular to a question answering model training, a question answering method and a device. Background Art
[0002] In primary school mathematics education, solving geometry problems requires not only students to have solid calculation skills, but also good graphic comprehension and spatial imagination. Therefore, the process of solving geometry problems is relatively complicated, especially in the current application context of large language models, which puts higher demands on the model's graphic comprehension and reasoning capabilities.
[0003] At present, the common method is to perform optical character recognition (OCR) on the images in the geometry questions and directly input the extracted text into the large language model, and use the reasoning ability of the large language model to infer the corresponding answer. Although the above method fully utilizes the advantages of the large language model in text reasoning, its effect is not ideal in solving geometry extraction. Summary of the invention
[0004] The present invention provides a question answering model training, a question answering method and a device to solve the defects in the prior art.
[0005] The present invention provides a question answering model training method, comprising the following steps: Acquire pre-training data and training data, wherein the pre-training data includes a first question image, a first question stem text in the first question image, and a first description text for describing a graphic area in the first question image, and the training data includes a second question image, and an answer text corresponding to the second question image; Based on the pre-training data, pre-training the initial model to obtain a pre-training model; Determining a predicted answer text corresponding to the second question image based on the pre-trained model; Based on the predicted answer text and the answer text, the pre-trained model is fine-tuned to obtain a question answering model.
[0006] According to a question answering model training method provided by the present invention, the initial model is pre-trained based on the pre-training data to obtain a pre-training model, including: Based on the initial model, determining a first predicted question stem text corresponding to the first question image and a first predicted description text corresponding to the first question image; Based on the difference between the first predicted question stem text and the first question stem text, and the difference between the first predicted description text and the first description text, the initial model is pre-trained to obtain the pre-trained model.
[0007] According to a question answering model training method provided by the present invention, the initial model is constructed based on a visual encoder and a language model; The determining, based on the initial model, a first predicted question stem text corresponding to the first question image and a first predicted description text corresponding to the first question image includes: Extracting visual features of the first topic image based on the visual encoder; Based on the language model, the visual features are applied to determine the first predicted topic text and the first predicted description text.
[0008] According to a question answering model training method provided by the present invention, the language model includes an attention layer, a fusion layer and a prediction layer, and the visual encoder is connected to the fusion layer; The determining the first predicted topic text and the first predicted description text based on the language model and applying the visual features includes: Extracting text features in the first title image based on the attention layer; Based on the fusion layer, the text feature and the visual feature are fused to obtain a fusion feature; Based on the prediction layer, applying the fusion feature, determining the first prediction topic text and the first prediction description text; Wherein, during the pre-training process of the initial model, the parameters of the attention layer and the prediction layer are frozen.
[0009] According to a question answering model training method provided by the present invention, determining the predicted answer text corresponding to the second question image based on the pre-trained model includes: Based on the pre-trained model, determine a second question stem text in the second question image and a second description text for describing a graphic area in the second question image; Based on the pre-trained model, the second question stem text and the second description text are applied to determine the predicted answer text corresponding to the second question image.
[0010] According to a question answering model training method provided by the present invention, the first description text is determined based on the graphic elements in the first question stem text and the text information in the graphic area.
[0011] The present invention also provides a method for answering questions, comprising the following steps: Determine the image of the question to be answered; Determine the answer text corresponding to the question image based on the question answering model; The question answering model is determined based on the question answering model training method described above.
[0012] The present invention also provides a question answering model training device, comprising the following modules: an acquisition unit, configured to acquire pre-training data and training data, wherein the pre-training data includes a first question image, a first question stem text in the first question image, and a first description text for describing a graphic area in the first question image, and the training data includes a second question image, and an answer text corresponding to the second question image; A pre-training unit, used to pre-train the initial model based on the pre-training data to obtain a pre-trained model; A prediction unit, configured to determine a predicted answer text corresponding to the second question image based on the pre-trained model; The training unit is used to fine-tune the pre-trained model based on the predicted answer text and the answer text to obtain a question answering model.
[0013] The present invention also provides a question answering device, comprising the following modules: A determination unit, used for determining the image of the question to be answered; An answering unit, used for determining an answer text corresponding to the question image based on the answering model; The question answering model is determined based on the question answering model training method described above.
[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any one of the above-described question answering model training methods or question answering methods.
[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements any of the above-described question answering model training methods or question answering methods.
[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned question answering model training methods or question answering methods.
[0017] The answering model training, answering method and device provided by the present invention improve the ability to understand the graphic area in the question and the ability to extract text information in the graphic area through pre-training, and fine-tune the pre-trained model through the training data in the training stage to make it better adapt to the answering task and be able to answer questions containing graphics, thus overcoming the defect that traditional solutions cannot accurately answer questions containing graphics. The trained answering model realizes effective answers to questions of any type, while ensuring the accuracy of the answers, and improving the answering efficiency, which provides benefits for students' learning efficiency and the optimization of the use experience of related learning equipment. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 It is a flowchart of the question answering model training provided by the present invention.
[0020] Figure 2 It is a schematic diagram of a geometry problem provided by the present invention.
[0021] Figure 3 It is a flowchart of the implementation of step 120 in the question answering model training method provided by the present invention.
[0022] Figure 4 It is a schematic diagram of the pre-training process provided by the present invention.
[0023] Figure 5 It is a flowchart of the implementation of step 130 in the question answering model training method provided by the present invention.
[0024] Figure 6 It is a schematic diagram of fine-tuning the pre-training model provided by the present invention.
[0025] Figure 7 It is a flow chart of the question answering method provided by the present invention.
[0026] Figure 8 It is a structural schematic diagram of the question answering model training device provided by the present invention.
[0027] Fig. 9 It is a structural schematic diagram of the question answering device provided by the present invention.
[0028] Fig.10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0030] At present, the common method is to perform optical character recognition (OCR) on the images in the geometry questions and directly input the extracted text into the large language model (hereinafter referred to as the "large model"), and use the reasoning ability of the large model to infer the corresponding answer. Although the above method fully utilizes the advantages of the large model in text reasoning, its effect is not ideal in solving geometry extraction, as follows: The large model does not fully understand the image information, and it is difficult for the large model to correctly identify the various shapes of the image. During the pre-training process, the large model mainly uses a large-scale text corpus to train the model's ability to predict the subsequent text based on the previous text. Therefore, the large model has a strong reasoning ability for text output. However, when answering geometry questions, the large model needs to understand not only the input question stem, but also the input question picture. Since the large model has only been exposed to a small amount of picture information, it is not accurate in identifying various graphic elements in the question picture.
[0031] In addition, it is difficult for the large model to align the graphic elements in the question image with the text in the question stem when answering questions. In the process of answering geometry questions, in order to better answer the questions, the large model needs to be able to understand the correspondence between the shapes mentioned in the question stem and the shapes in the picture. For example, if the question stem mentions △ABC, the model needs to be able to associate it with the area corresponding to △ABC in the picture, so that it can correctly infer and answer the question.
[0032] Figure 1 It is a flow chart of the answering model training method provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 , step 130 and step 140 .
[0033] Step 110, obtaining pre-training data and training data, the pre-training data including the first question image, the first question stem text in the first question image, and the first description text for describing the graphic area in the first question image, the training data including the second question image, and the answer text corresponding to the second question image.
[0034] Specifically, before model training, it is necessary to first obtain training data. Specifically in the embodiment of the present invention, in order to ensure the generalization performance of the model, as well as the precision and accuracy in the question-answering task, the overall training process can be divided into two stages, namely the pre-training stage and the training stage. The pre-training stage improves the model's ability to understand the graphic area in the question and the ability to extract text information in the graphic area, so that the model can accurately answer questions without inputting additional information. On this basis, downstream fine-tuning is performed to fine-tune the model parameters to make it better suitable for downstream question-answering tasks, thereby improving the model's accuracy in answering questions without affecting the generalization performance of the model.
[0035] Based on this, in an embodiment of the present invention, it is first necessary to obtain training samples of each stage, that is, sample data of the pre-training stage, that is, pre-training data, and sample data of the training stage, that is, training data; because the pre-training stage needs to understand the question graphic area, extract text information in the graphic area, and align the extracted text information with the question stem text, etc., therefore, the sample data of the pre-training stage needs to include data that enables the model to learn in the above multiple aspects, for example, it may include a question image, the first question stem text in the first question image, and the first description text used to describe the graphic area in the first question image. The first description text can be understood as text information used to describe the graphic area, which can be a contour description corresponding to the graphic element mentioned in the first question stem text in the graphic area.
[0036] Figure 2 It is a schematic diagram of a geometry problem provided by the present invention, such as Figure 2 As shown, the first question text is "3. Calculate the area of the shaded part in the figure below", and the graphic element in the first question text is "shaded part". The position of the graphic element in the graphic area is marked and converted into the first description text "In the question, the "shaded part" is composed of a triangle with a height of 10 cm [(75,103), (130,301), (75,301)] and a triangle with a height of 10 cm [(405,103), (405,301), (130,301)]".
[0037] Correspondingly, the sample data in the training phase includes data that enables the model to be better adapted to the question answering task, and can be used to answer the question based on the second question image to obtain the corresponding answer text, for example, it can include the second question image and the answer text corresponding to the second question image. The first question image and the second question image here can be the same or different.
[0038] Among them, in the training samples of each stage, the question images (first question graphics and second question graphics) can be obtained by image acquisition equipment on existing paper questions, or can be obtained by searching, crawling, or directly downloaded / intercepted from a specific test website, or can be found in a question bank, and the embodiments of the present invention do not make specific limitations on this.
[0039] The question stem text may be obtained by manual annotation, or by performing OCR recognition on the question graphic, or by manual input, which is not specifically limited in the embodiment of the present invention.
[0040] The answer text can be annotated by experienced experts, teachers, etc., or it can be obtained together with the question text or question image when the corresponding question text or question image is obtained, that is, the question and its corresponding standard answer are directly obtained from a question bank or a specific website that carries the standard answer as the answer text. It can also be obtained by extracting the answer from paper materials such as textbooks and teaching supplementary materials that carry the test answers. The embodiments of the present invention do not make specific limitations on this.
[0041] Step 120: pre-train the initial model based on the pre-training data to obtain a pre-trained model.
[0042] Specifically, after obtaining the pre-training data in step 110, the model can be pre-trained according to the pre-training data to obtain a pre-trained model, that is, a pre-trained model.
[0043] It is worth noting that before using pre-training data to pre-train the model, it is necessary to first determine the initial model. Considering that the traditional solution can only answer pure text questions, the input and output of the model do not involve images, and the images cannot be processed, so that questions containing graphics cannot be effectively answered. In an embodiment of the present invention, in order to achieve automatic answering of questions containing graphics, such as connection questions, clock questions, geometry questions, etc., a visual encoder capable of processing images can be added when constructing the initial model, so as to perform image encoding to understand the semantic information contained therein, thereby achieving answering of questions with pictures.
[0044] Based on this, in an embodiment of the present invention, when constructing the initial model, a visual encoder can be added, and on this basis, a language model, such as a large language model, a cognitive model, etc., can be combined to construct an initial model before pre-training.
[0045] It can be understood that the purpose of pre-training the model using pre-training data is to enhance the model's ability to understand the graphic area in the question and extract text information in the graphic area. Therefore, the entire pre-training process can be regarded as a prediction of the model's extraction of the question stem text and the text information in the graphic area aligned with the question stem text. The input and output of the two are not the same. Therefore, in an embodiment of the present invention, the pre-training process can be segmented and subdivided into two stages, namely, the question intervention training stage and the description pre-training stage.
[0046] Among them, the question intervention training stage uses the first question image-the first question stem text to train the model, so that the model can predict the question stem text according to the input question image, and adjust the model parameters by comparing the predicted question stem text with the first question stem text, so that the question stem text extracted by the model after parameter adjustment for the input question image can be as close to the first question stem text as possible, or consistent with the first question stem text, so that the model has the ability to extract question stems, thereby obtaining a trained model.
[0047] The description pre-training stage uses the first question image-first description text to train the model, so that the model can predict the text information in the graphic area aligned with the question text for the input first question image, and output the corresponding predicted description text. By comparing the predicted description text with the first description text, the model parameters are adjusted so that the predicted description text output by the model after parameter adjustment for the input first question image can be as close to the first description text as possible, or consistent with the first description text, so that the model has the ability to understand the graphic area in the question and extract the text information in the graphic area, and finally a trained model can be obtained.
[0048] Step 130: Determine the predicted answer text corresponding to the second question image based on the pre-trained model.
[0049] Specifically, after the pre-training in step 120 is completed and the pre-training model is obtained, the downstream task can be performed based on the training data according to the pre-training model to determine the predicted answer text corresponding to the training data.
[0050] It can be understood that after the pre-training is completed, the training phase can be entered to use the training data to train the pre-training model to obtain the final answering model. In this process, the pre-training model can first be applied to process the training data to obtain a second description text for describing the graphic area in the second question image, and then the questions can be answered on this basis to obtain the corresponding answer text, that is, the predicted answer text.
[0051] Step 140: Based on the predicted answer text and the answer text, fine-tune the pre-trained model to obtain a question answering model.
[0052] Specifically, after obtaining the predicted answer text predicted by the pre-trained model through the above process, the answer text can be predicted based on it, and the pre-trained model can be fine-tuned based on the training data to obtain the answer model.
[0053] It can be understood that after the above prediction process, after obtaining the results of the pre-trained model's prediction output for the training data, the loss of the model in the above prediction process can be measured through the model prediction results and the labels in the training data, so as to fine-tune the model parameters according to the loss and finally obtain a trained answering model.
[0054] Specifically, the predicted answer text and the answer text can be used to determine the loss of the pre-trained model in the prediction task during the training phase, and then the parameters of the pre-trained model can be iterated based on this loss to obtain the final trained answering model.
[0055] The above process is essentially to measure the loss of the model in the prediction task by the difference between the output of the model in the prediction task and the label of the input image. This loss can reflect the deficiency of the model in the prediction task. Then the parameters of the model can be fine-tuned according to this deficiency, so that the predicted answer text output by the model for the input question image after parameter adjustment can be closer to the label in the training data, that is, the predicted answer text output by the pre-trained model for the input second question image after parameter adjustment can be as close to the answer text as possible. Through continuous prediction output and parameter fine-tuning, the prediction result output by the model can be infinitely close to the label, or completely consistent with the label. The model at this time is the final model, that is, the answering model.
[0056] The question-answering model training method provided by the present invention improves the ability to understand the graphic area in the question and the ability to extract text information in the graphic area through pre-training, and fine-tunes the pre-trained model through the training data in the training stage to make it better adapt to the question-answering task and can answer questions containing graphics, thus overcoming the defect that the traditional solution cannot accurately answer questions containing graphics. The trained question-answering model realizes effective answers to questions of any type, and improves the answering efficiency while ensuring the accuracy of the answers, thereby providing benefits for students' learning efficiency and the optimization of the use experience of related learning equipment.
[0057] Based on the above embodiments, Figure 3 is a flowchart of the implementation of step 120 in the question answering model training method provided by the present invention, such as Figure 3As shown, step 120 pre-trains the initial model based on the pre-training data to obtain a pre-training model, including: Step 121: Determine, based on the initial model, a first predicted question stem text corresponding to the first question image and a first predicted description text corresponding to the first question image; Step 122: Based on the difference between the first predicted question stem text and the first question stem text, and the difference between the first predicted description text and the first description text, pre-train the initial model to obtain a pre-trained model.
[0058] Specifically, in step 120, the process of training the initial model according to the pre-training data to obtain the pre-training model may specifically include: First, the initial model can be used to determine the first predicted question stem text corresponding to the first question image and the first predicted description text corresponding to the first question image; wherein the first predicted description text is the text information predicted by the initial model based on the first question image and used to describe the graphic area in the first question image.
[0059] Next, the loss of the initial model in the pre-training stage is determined by the difference between the first predicted question stem text and the first question stem text, and the difference between the first predicted description text and the first description text, and the parameters are adjusted according to the loss to obtain the pre-trained model.
[0060] Based on any of the above embodiments, the initial model is constructed based on the visual encoder and the language model; Determining, based on the initial model, a first predicted question stem text corresponding to the first question image and a first predicted description text corresponding to the first question image, including: Extracting visual features of the first topic image based on the visual encoder; Based on the language model, visual features are applied to determine the first predicted question stem text and the first predicted description text.
[0061] Specifically, the input and output of the language model do not involve images and cannot process images, resulting in an inability to effectively answer questions containing graphics. In an embodiment of the present invention, in order to automatically answer questions containing graphics, such as connection questions, clock questions, geometry questions, etc., a visual encoder capable of processing images can be added when constructing the initial model to perform image encoding in order to understand the semantic information contained therein, thereby answering questions with graphics.
[0062] Based on this, in an embodiment of the present invention, when constructing the initial model, a visual encoder can be added, and on this basis, a language model, such as a large language model, a cognitive model, etc., can be combined to construct an initial model before pre-training.
[0063] Specifically, an embodiment of the present invention inputs the first question image into a visual encoder, which extracts visual features (that is, converts graphic information in the first question graphic into visual tokens), and then uses the understanding ability of the language model to apply the visual features to determine the first predicted question stem text and the first predicted description text.
[0064] Based on any of the above embodiments, the language model includes an attention layer, a fusion layer, and a prediction layer, and the visual encoder is connected to the fusion layer; Based on the language model, visual features are applied to determine the first predicted topic text and the first predicted description text, including: Based on the attention layer, extract the text features in the first question image; Based on the fusion layer, the text features and visual features are fused to obtain fusion features; Based on the prediction layer, the fusion features are applied to determine the first prediction topic text and the first prediction description text; Among them, during the pre-training process of the initial model, the parameters of the attention layer and the prediction layer are frozen.
[0065] Specifically, the visual encoder is connected to the fusion layer of the language model, and then the fusion layer of the language model can fuse the visual features extracted by the visual encoder and the text features extracted by the attention layer, that is, fuse the image information and text information in the first question image, and the prediction layer determines the first predicted question stem text and the first predicted description text based on the fused features.
[0066] That is to say, there are two training goals in the pre-training stage. One goal is to extract the question stem text, that is, to extract the text information in the first question image, so as to train the model's ability to understand graphic text. The other goal is to describe the graphic information in the first question image (such as the shape and specifications of the graphic). Based on the above two goals, the pre-training model can better understand the graphic information in the question image, and the fusion layer fuses the visual features and text features, that is, the visual features and text features are represented in the same embedding space, and the visual features and text features share the same loss function, thereby ensuring that the visual features and text features can be aligned to the same feature space, thereby aligning the first predicted description text with the first predicted question stem text. Among them, in order to maintain the language model's own ability to answer questions, only the visual encoder and the fusion layer are trained during the pre-training process, that is, the parameters of the remaining layers (attention layer and prediction layer) are frozen.
[0067] Figure 4 It is a schematic diagram of the pre-training process provided by the present invention, such as Figure 4As shown in the figure, ConvNext-base is used as the visual encoder, and the language model includes an attention layer (Self-attention), a fusion layer (Cross-attention) and a prediction layer (FFN), ConvNext-base and Cross-attention. The pre-training process is as follows: ConvNext-base is used to extract the visual features of the first question image, and the visual features are input into Cross-attention. Self-attention is used to extract the text features of the first question image, and the text features are input into Cross-attention. Cross-attention fuses the visual features and text features to obtain fused features. FFN determines the first predicted question stem text and the first predicted description text based on the fused features.
[0068] Based on any of the above embodiments, Figure 5 is a flowchart of the implementation of step 130 in the question answering model training method provided by the present invention, such as Figure 5 As shown, step 130 determines the predicted answer text corresponding to the second question image based on the pre-trained model, including: Step 131: Determine, based on the pre-trained model, a second question stem text in the second question image and a second description text for describing a graphic area in the second question image; Step 132: Based on the pre-trained model, the second question stem text and the second description text are applied to determine the predicted answer text corresponding to the second question image.
[0069] Specifically, in step 130, determining the predicted answer text corresponding to the second question image according to the pre-trained model may specifically include: First, the second question stem text in the second question image and the second description text for describing the graphic area in the first question image can be determined through a pre-trained model, that is, based on the second question image, the pre-trained model is applied to predict it to obtain the second question stem text and the second description text.
[0070] Here, specifically, the second question image may be taken as input and input into the pre-trained model, so that the pre-trained model predicts the input second question image to obtain the second question stem text and the second description text.
[0071] Then, based on the predicted second question stem text and second description text, the corresponding predicted answer text can be determined through the pre-trained model, that is, based on the second question stem text and the second description text, the predictive training model is applied to answer the question, so as to obtain the predicted answer text corresponding to the second question image.
[0072] It can be seen that before answering the question, the pre-trained model first determines the second description text used to describe the graphic area in the second question image, so that the pre-trained model can explicitly perceive the graphic information through the second description text during the question answering reasoning process, and more effectively understand the graphic information and make correct answers. In addition, when predicting the second description text, the pre-trained model can further improve its ability to understand graphic information, so that the pre-trained model understands the correspondence between the shape name mentioned in the question stem and the specific position in the graphic, so as to better answer the question. Furthermore, before answering the question, the pre-trained model first determines the second description text used to describe the graphic area in the second question image, and does not increase the burden of the pre-trained model during reasoning, so that the pre-trained model can use the same input as the conventional model to make a more accurate answer.
[0073] Figure 6 is a schematic diagram of fine-tuning the pre-training model provided by the present invention, such as Figure 6 As shown in FIG. 1 , the pre-trained model includes a visual encoder (ConvNext-base) and a language model. The language model includes an attention layer (Self-attention), a fusion layer (Cross-attention), and a prediction layer (FFN). The specific fine-tuning process is as follows: the visual features of the second question image are extracted through ConvNext-base, and the visual features are input into Cross-attention. Self-attention extracts the text features of the second question image, and the text features are input into Cross-attention. Cross-attention fuses the visual features and text features to obtain fusion features. FFN determines the second predicted question stem text and the second predicted description text based on the fusion features, and obtains the predicted answer text based on the second predicted question stem text and the second predicted description text. Based on the difference between the predicted answer text and the answer text, the parameters of the pre-trained model are adjusted. In the fine-tuning process, the parameters of all layers of the pre-trained model are updated, that is, the parameters of the visual encoder, the attention layer, the fusion layer, and the prediction layer are updated.
[0074] Based on any of the above embodiments, the first description text is determined based on the graphic elements in the first question stem text and the text information in the graphic area.
[0075] Specifically, the graphic elements in the first question text can be understood as entities used to indicate graphics in the first question text. For example, if the first question text is "3. Calculate the area of the shaded part in the following figure", the "shaded part" indicates the figure, so the "shaded part" is regarded as a graphic element. Graphic elements include points, lines, circles, triangles, rectangles, parallelograms, trapezoids, sectors, cubes, cylinders, and cones.
[0076] In addition, some graphical elements may appear in the graphics area, such as Figure 2 The graphic area includes the text "15cm" and "10cm", which does not appear in the question text. However, the text information describes that the graphic area also includes "a rectangle that is 15cm long and 10cm wide", that is, the text information is used to describe the graphic information not mentioned in the question text.
[0077] Therefore, an embodiment of the present invention determines the first description text by combining the graphic elements in the first question text and the text information in the graphic area, so as to comprehensively describe the graphic information in the first question image, thereby helping the model to more accurately understand the graphic information in the image and give accurate answers.
[0078] like Figure 2 As shown, according to the graphic element "shaded part" in the text of the first question, it can be determined that the first description text includes "the shaded part is composed of a triangle with a height of 10 cm [(75,103),(130,301),(75,301)] and a triangle with a height of 10 cm [(405,103),(405,301),(130,301)]". According to the text "15cm" and "10cm" in the graphic area, it can be determined that the first description text also includes "the graphic area also includes a rectangle with a length of 15 cm and a width of 10 cm [(75,103),(405,103),(405,301),(75,301)]".
[0079] Based on any of the above embodiments, Figure 7 It is a flow chart of the answering method provided by the present invention, such as Figure 7 As shown, the method includes: Step 710: determining the image of the question to be answered; Step 720: Determine the answer text corresponding to the question image based on the question answering model; The question answering model is determined based on the question answering model training method described in any of the above embodiments.
[0080] Specifically, before answering questions, you first need to determine the questions that need to be answered automatically, that is, the questions to be answered. Specifically, you can obtain the image of the question to be answered. The question image can be obtained by image acquisition equipment on existing paper questions, or by searching, crawling, or directly downloading / intercepting from a specific question website. It can also be found in a question library. The embodiment of the present invention does not make specific limitations on this.
[0081] Here, the question image to be answered may contain one question or multiple questions. When it contains multiple questions, the entire question image can be cropped into multiple blocks, each of which contains one question. In this way, when the answering model is subsequently applied to answer questions, it is convenient for the answering model to process each question and output the corresponding answer text.
[0082] It is worth noting that the question image to be answered here can be one or more. In the case of multiple question images, each question image can be answered to obtain the answer text corresponding to each question image.
[0083] Furthermore, after determining the question image to be answered, you can answer the question it contains to obtain the corresponding answer text, that is, you can apply a pre-trained answering model to answer the question and obtain the answer text corresponding to the question image. Specifically here, the question image can be input into the answering model so that the answering model understands the input question image and answers it, thereby obtaining the predicted output answer text, that is, the answer text corresponding to the question image.
[0084] Among them, the answering model can be trained based on the following steps: obtain pre-training data and training data, the pre-training data includes a first question image, a first question stem text in the first question image, and a first description text for describing the graphic area in the first question image, and the training data includes a second question image, and an answer text corresponding to the second question image; based on the pre-training data, pre-train the initial model to obtain a pre-training model; based on the pre-training model, determine the predicted answer text corresponding to the second question image; based on the predicted answer text and the answer text, fine-tune the pre-training model to obtain the answering model.
[0085] The question answering method provided by the present invention uses the question answering model trained by the question answering model training method to answer the question image and obtain the answer text corresponding to the question image, thereby overcoming the defect that the traditional scheme cannot accurately answer the questions containing graphics. The trained question answering model realizes the effective answering of questions of any type, and improves the answering efficiency while ensuring the accuracy of the answer, which provides benefits for the students' learning efficiency and the optimization of the use experience of related learning equipment.
[0086] The question answering model training device provided by the present invention is described below. The question answering model training device described below and the question answering model training method described above can be referenced to each other.
[0087] Based on any of the above embodiments, Figure 8 Schematic diagram of the structure of the answering model training device provided by the present invention. Figure 8 As shown, the device comprises: The acquisition unit 810 is used to acquire pre-training data and training data, wherein the pre-training data includes a first question image, a first question stem text in the first question image, and a first description text for describing a graphic area in the first question image, and the training data includes a second question image, and an answer text corresponding to the second question image; A pre-training unit 820 is used to pre-train the initial model based on the pre-training data to obtain a pre-training model; A prediction unit 830, configured to determine a predicted answer text corresponding to the second question image based on the pre-trained model; The training unit 840 is used to fine-tune the pre-trained model based on the predicted answer text and the answer text to obtain a question answering model.
[0088] Based on any of the above embodiments, the initial model is pre-trained based on the pre-training data to obtain a pre-training model, including: Based on the initial model, determining a first predicted question stem text corresponding to the first question image and a first predicted description text corresponding to the first question image; Based on the difference between the first predicted question stem text and the first question stem text, and the difference between the first predicted description text and the first description text, the initial model is pre-trained to obtain a pre-trained model.
[0089] Based on any of the above embodiments, the initial model is constructed based on the visual encoder and the language model; Determining, based on the initial model, a first predicted question stem text corresponding to the first question image and a first predicted description text corresponding to the first question image, including: Extracting visual features of the first topic image based on the visual encoder; Based on the language model, visual features are applied to determine the first predicted question stem text and the first predicted description text.
[0090] Based on any of the above embodiments, the language model includes an attention layer, a fusion layer, and a prediction layer, and the visual encoder is connected to the fusion layer; Based on the language model, visual features are applied to determine the first predicted topic text and the first predicted description text, including: Based on the attention layer, extract the text features in the first question image; Based on the fusion layer, the text features and visual features are fused to obtain fusion features; Based on the prediction layer, the fusion features are applied to determine the first prediction topic text and the first prediction description text; Among them, during the pre-training process of the initial model, the parameters of the attention layer and the prediction layer are frozen.
[0091] Based on any of the above embodiments, determining the predicted answer text corresponding to the second question image based on the pre-trained model includes: Based on the pre-trained model, determine a second question stem text in the second question image and a second description text for describing the graphic area in the second question image; Based on the pre-trained model, the second question stem text and the second description text are applied to determine the predicted answer text corresponding to the second question image.
[0092] Based on any of the above embodiments, the first description text is determined based on the graphic elements in the first question stem text and the text information in the graphic area.
[0093] The question answering device provided by the present invention is described below. The question answering device described below and the question answering method described above can be referenced to each other.
[0094] Based on any of the above embodiments, Fig. 9 Schematic diagram of the structure of the answering device provided by the present invention. Fig. 9 As shown, the device comprises: A determination unit 910, configured to determine a question image to be answered; The answering unit 920 is used to determine the answer text corresponding to the question image based on the answering model; The question answering model is determined based on the question answering model training method described in any of the above embodiments.
[0095] Fig.10 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute the answer model training method, the method comprising: obtaining pre-training data and training data, the pre-training data comprising a first question image, a first question stem text in the first question image and a first description text for describing a graphic area in the first question image, the training data comprising a second question image and an answer text corresponding to the second question image; pre-training the initial model based on the pre-training data to obtain a pre-training model; determining the predicted answer text corresponding to the second question image based on the pre-training model; fine-tuning the pre-training model based on the predicted answer text and the answer text to obtain a question answering model.
[0096] Or, execute a question answering method, the method comprising: determining a question image to be answered; based on a question answering model, determining an answer text corresponding to the question image; the question answering model is determined based on the question answering model training method as described above.
[0097] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0098] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the question answering model training method provided by the above methods, and the method includes: obtaining pre-training data and training data, the pre-training data including a first question image, a first question stem text in the first question image, and a first description text for describing a graphic area in the first question image, the training data including a second question image, and an answer text corresponding to the second question image; based on the pre-training data, pre-training an initial model to obtain a pre-training model; based on the pre-training model, determining a predicted answer text corresponding to the second question image; based on the predicted answer text and the answer text, fine-tuning the pre-training model to obtain a question answering model.
[0099] Or, execute a question answering method, the method comprising: determining a question image to be answered; based on a question answering model, determining an answer text corresponding to the question image; the question answering model is determined based on the question answering model training method as described above.
[0100] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the question answering model training method provided by the above-mentioned methods, the method comprising: obtaining pre-training data and training data, the pre-training data comprising a first question image, a first question stem text in the first question image, and a first description text for describing a graphic area in the first question image, the training data comprising a second question image, and an answer text corresponding to the second question image; based on the pre-training data, pre-training an initial model to obtain a pre-training model; based on the pre-training model, determining a predicted answer text corresponding to the second question image; based on the predicted answer text and the answer text, fine-tuning the pre-training model to obtain a question answering model.
[0101] Or, execute a question answering method, the method comprising: determining a question image to be answered; based on a question answering model, determining an answer text corresponding to the question image; the question answering model is determined based on the question answering model training method as described above.
[0102] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0103] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A question answering model training method, characterized in that: include: Acquire pre-training data and training data, wherein the pre-training data includes a first question image, a first question stem text in the first question image, and a first description text for describing a graphic area in the first question image, and the training data includes a second question image, and an answer text corresponding to the second question image; Based on the pre-training data, pre-training the initial model to obtain a pre-training model; Determining a predicted answer text corresponding to the second question image based on the pre-trained model; Based on the predicted answer text and the answer text, the pre-trained model is fine-tuned to obtain a question answering model.
2. The question answering model training method according to claim 1, characterized in that: The pre-training of the initial model based on the pre-training data to obtain the pre-training model includes: Based on the initial model, determining a first predicted question stem text corresponding to the first question image and a first predicted description text corresponding to the first question image; Based on the difference between the first predicted question stem text and the first question stem text, and the difference between the first predicted description text and the first description text, the initial model is pre-trained to obtain the pre-trained model.
3. The question answering model training method according to claim 2, characterized in that: The initial model is constructed based on the visual encoder and the language model; The determining, based on the initial model, a first predicted question stem text corresponding to the first question image and a first predicted description text corresponding to the first question image includes: Extracting visual features of the first topic image based on the visual encoder; Based on the language model, the visual features are applied to determine the first predicted topic text and the first predicted description text.
4. The question answering model training method according to claim 3, characterized in that: The language model includes an attention layer, a fusion layer and a prediction layer, and the visual encoder is connected to the fusion layer; The determining the first predicted topic text and the first predicted description text based on the language model and applying the visual features includes: Based on the attention layer, extracting text features in the first title image; Based on the fusion layer, the text feature and the visual feature are fused to obtain a fusion feature; Based on the prediction layer, applying the fusion feature, determining the first prediction topic text and the first prediction description text; Wherein, during the pre-training process of the initial model, the parameters of the attention layer and the prediction layer are frozen.
5. The answer model training method according to any one of claims 1 to 4, characterized in that: The step of determining the predicted answer text corresponding to the second question image based on the pre-trained model includes: Based on the pre-trained model, determine a second question stem text in the second question image and a second description text for describing a graphic area in the second question image; Based on the pre-trained model, the second question stem text and the second description text are applied to determine the predicted answer text corresponding to the second question image.
6. The answer model training method according to any one of claims 1 to 4, characterized in that: The first description text is determined based on the graphic elements in the first question stem text and the text information in the graphic area.
7. A method for answering questions, characterized in that: include: Determine the image of the question to be answered; Determine the answer text corresponding to the question image based on the question answering model; The question answering model is determined based on the question answering model training method as described in any one of claims 1 to 6.
8. A question answering model training device, characterized in that: include: an acquisition unit, configured to acquire pre-training data and training data, wherein the pre-training data includes a first question image, a first question stem text in the first question image, and a first description text for describing a graphic area in the first question image, and the training data includes a second question image, and an answer text corresponding to the second question image; A pre-training unit, used to pre-train the initial model based on the pre-training data to obtain a pre-trained model; A prediction unit, configured to determine a predicted answer text corresponding to the second question image based on the pre-trained model; The training unit is used to fine-tune the pre-trained model based on the predicted answer text and the answer text to obtain a question answering model.
9. A question answering device, characterized in that: include: A determination unit, used for determining the image of the question to be answered; An answering unit, used for determining an answer text corresponding to the question image based on the answering model; The question answering model is determined based on the question answering model training method as described in any one of claims 1 to 6.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for answering questions as described in any one of claims 1 to 6 or the method for answering questions as described in claim 7 is implemented.
11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the question answering method according to any one of claims 1 to 6, or the question answering method according to claim 7.
12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the question answering method according to any one of claims 1 to 6, or the question answering method according to claim 7.