Homework grading method and apparatus

Abstract

The present disclosure relates to a homework grading method and apparatus, the homework grading method comprising: acquiring a question image, and performing text recognition on the question image to generate a text recognition result and a target matrix, the target matrix being formed on the basis of the sequence of answer texts comprised in the question image; using the text recognition result and the question image as inputs of a pre-trained multi-modal model to obtain an output result, the output result comprising a text to be graded of a target question in the question image and the question type of the target question; determining a grading mode corresponding to the question type, and, on the basis of said text, obtaining in the grading mode a reference text; and comparing the reference text with the answer text corresponding to the target question in the target matrix, so as to generate a grading result of the target question.

Description

Homework correction method and device Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method and device for grading homework. Background Art

[0002] With the rapid development of computer technology, artificial intelligence (AI) is being widely used in education. For example, it can automatically grade user-captured images and provide feedback to users, allowing them to instantly understand their homework progress. The current grading strategy involves searching a database for identical questions and then comparing the user's answers with the database to determine correct answers. Summary of the Invention

[0003] The present disclosure provides a method and device for grading homework.

[0004] According to one aspect of the present disclosure, a method for grading homework is provided, comprising: obtaining a question image, and performing text recognition on the question image to generate a text recognition result and a target matrix, wherein the target matrix is ​​generated based on an answer text included in the question image; using the text recognition result and the question image as inputs of a pre-trained multimodal model to obtain an output result, wherein the output result includes a text to be graded of a target question in the question image and a question type of the target question; determining a grading mode corresponding to the question type, and obtaining a reference text based on the text to be graded under the grading mode; comparing the reference text with the answer text corresponding to the target question in the target matrix to generate a grading result for the target question.

[0005] According to another aspect of the present disclosure, a homework grading device is provided, comprising: an acquisition unit, configured to acquire a question image, and perform text recognition on the question image to generate a text recognition result and a target matrix, wherein the target matrix is ​​formed based on an answer text included in the question image; a recognition unit, configured to use the text recognition result and the question image as inputs of a pre-trained multimodal model to obtain an output result, wherein the output result includes a text to be graded of a target question in the question image and a question type of the target question; a correction unit, configured to determine a correction mode corresponding to the question type, and obtain a reference text based on the text to be graded under the correction mode; and a generation unit, configured to compare the reference text with the answer text corresponding to the target question in the target matrix to generate a correction result for the target question.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory coupled to the at least one processor, wherein the at least one processor is configured to read a computer program stored in the at least one memory to execute the above-mentioned homework grading method.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. When the computer instructions are executed by a processor of a computer, the computer is caused to perform the above-mentioned homework grading method.

[0008] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program. When the computer program is executed by a processor of a computer, the computer is caused to execute the above-mentioned homework grading method. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0010] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] FIG1 is a flow chart of a method for grading homework provided by an embodiment of the present disclosure;

[0012] FIG2 is a schematic diagram of a topic image provided by an embodiment of the present disclosure;

[0013] FIG3 is a detailed flow chart of step S102 in the homework grading method provided by an embodiment of the present disclosure;

[0014] FIG4 is a detailed flowchart of step S103 in the homework grading method provided by an embodiment of the present disclosure;

[0015] FIG5 is a flow chart of another homework grading method provided by an embodiment of the present disclosure;

[0016] FIG6 is a schematic structural diagram of a homework correction device provided by an embodiment of the present disclosure;

[0017] FIG7 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0019] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0020] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] Currently, there are the following problems with homework grading strategies.

[0024] (1) The problem of database needing real-time updating: The current grading strategy requires searching for the same questions in the database and correcting the answers by comparing the user's answers with the answers in the database. Therefore, a large amount of answer data is needed to support the database. At the same time, it is also necessary to collect the latest questions and update the input answers in real time. However, updating the answer database will consume a lot of manpower and material resources.

[0025] (2) Problems with limited scope of problem solving: Currently, the accuracy rate for single, simple answers is relatively high. However, for complex and diverse answers, or those that cannot be described in words, such as drawing questions and connecting lines questions, there is less support, or the answers are not exactly the same as those in the database, resulting in low accuracy.

[0026] (3) Problems where problem-solving strategies need to be updated in real time: For new question types that are different from those in the question bank, or variants of a certain question type, it is necessary to design corresponding problem-solving process strategies to support the question type.

[0027] In response to at least one of the above technical problems, the present application provides a homework grading method. After obtaining the image of the question to be graded, the question image is recognized to obtain the recognition result and a matrix composed of the order of the answer text. Through the understanding ability related to the multimodal model, logical reasoning is performed based on the question image and the recognition result to obtain the problem-solving result. This multimodal method also has a certain reasoning ability for new question types and is less affected by the scope of the problem-solving, effectively improving the accuracy of the problem-solving. Subsequently, the answers in the matrix are graded based on the problem-solving results, and the correction accuracy is also relatively high, which makes it easy for users to immediately understand the answer situation and further improves the user experience. Detailed explanation is given through the following embodiments.

[0028] Specifically, the homework grading method can be executed by a terminal or server. Specifically, the terminal or server can grade the question images using multiple machine learning models that assist in grading. The execution entity of the training methods of the multiple machine learning models that assist in grading can be the same as or different from the execution entity of the homework grading method.

[0029] For example, in one application scenario, the server trains a variety of machine learning models for auxiliary correction. The terminal obtains the trained machine learning models for auxiliary correction from the server, and the terminal corrects the title image using the trained machine learning models for auxiliary correction. The title image may be an image captured by the terminal. Alternatively, the title image is obtained by the terminal from other devices. Alternatively, the title image is an image obtained by the terminal after performing image processing on a preset image, and the preset image may be captured by the terminal, or the preset image may be obtained by the terminal from other devices. Other devices are not specifically limited here.

[0030] In another application scenario, the server trains multiple machine learning models for assisting with grading. Furthermore, the server uses these trained machine learning models to grade the question image. The server acquires the question image in a manner similar to the terminal's acquisition of the question image described above, and will not be further elaborated here.

[0031] In another application scenario, the terminal trains multiple machine learning models for assisting in correction. Further, the terminal corrects the question image using the trained multiple machine learning models for assisting in correction.

[0032] It is understandable that the various machine learning model training methods and homework correction methods for auxiliary correction provided by the embodiments of the present disclosure are not limited to the several possible scenarios described above.

[0033] FIG1 is a flow chart of a method for grading homework provided by an embodiment of the present disclosure. The method can be applied to a homework grading scenario and can be executed by a terminal. For example, in one possible scenario, a user terminal photographs the homework to be graded to obtain a title image, and uploads the title image to the terminal. The terminal grades the title image and feeds back the grading results to the user terminal. The method specifically includes the following steps S101 to S104 as shown in FIG1 .

[0034] In step S101 , a question image is acquired, and text recognition is performed on the question image to generate a text recognition result and a target matrix.

[0035] The target matrix is ​​composed of the order of the answer texts included in the question image.

[0036] It is understandable that the question image is obtained, for example, by using a camera configured for the learning device to photograph the user's homework to obtain the question image, and the question image includes at least one question, and the question can be at least one type of question among a drawing question, a connection question, an application question, a fill-in-the-blank question, and a judgment question. It is understandable that the question image can be an image corrected from the initial captured image, or the question image is the initial captured image, which is not limited here. The following embodiment takes the question image as an example of a corrected image to illustrate a correction process, and other available correction methods are not described in detail here. The correction process is: select a homework paper area from the captured image, the homework paper area refers to the area containing the question and the answer, which can be understood as the area obtained after removing the background part of the image; use the perspective transformation method to transform the homework paper area to obtain a neat question image, and the question image can be understood as an image including the question and the answer.

[0037] As can be understood, after obtaining the question image, text recognition is performed on the question image to obtain a text recognition result that includes the complete content of the question image. At the same time, during the recognition process, all the answer texts involved are matrixed to obtain a target matrix. The target matrix is ​​composed of the answer texts according to their position order in the question image, so as to facilitate the precise location of the answer to the question during the subsequent correction process. This is explained in detail through the following embodiments.

[0038] Optionally, in step S101 , text recognition is performed on the title image to generate a text recognition result and a target matrix, which can be specifically achieved through the following steps.

[0039] Detect text lines in the question image to obtain multiple text boxes that select each text line; determine the first position of a target text box containing answer text among the multiple text boxes in the question image, and perform a matrix operation on the first position and the answer text to generate a target matrix; perform text recognition on the multiple text boxes to generate a text recognition result.

[0040] It is understandable that the detection of a preset number of text lines in the question image can be performed, specifically, single-line detection or multi-line detection. Single-line detection refers to using a line of text as the detection unit to obtain a text box that selects each text line. In this case, each text box selects a line of text. Multi-line detection refers to using multiple lines of text as the detection unit to obtain a text box that selects multiple lines of text. Subsequently, single-line detection can be continued based on the text box, and the specific number of text lines detected is not limited. After obtaining multiple text boxes, the target text box containing the answer text is determined from the multiple text boxes. In one possible case, only one text box containing the answer text is detected from the question image. In another possible case, the target text box and the remaining text boxes containing only the question text are detected from the question image. Subsequently, the first position of the target text box in the question image is determined, the first position is converted into a row and column position, and the answer text included in the target text box is associated with the row and column position. For example, the answer text is directly used as the content at the row and column position to generate a target matrix. In addition, after obtaining multiple text boxes, the first recognition model is used to perform text recognition on the multiple text boxes to obtain a complete text recognition result. The first recognition model can use an optical character recognition method (Optical Character Recognition, OCR). Other methods that can achieve text recognition are not limited here and can be determined according to user needs.

[0041] The text lines include printed lines and / or handwritten lines.

[0042] Optionally, the above-mentioned detecting of text lines in the title image to obtain multiple text boxes for selecting each text line can be specifically achieved through the following steps.

[0043] The printed line and the handwritten line in the title image are detected to obtain a printed frame that selects the printed line and a handwritten frame that selects the handwritten line; the printed frame and the handwritten frame located on the same text line are merged to obtain a mixed frame; wherein the multiple text frames include the unmerged printed frame, the unmerged handwritten frame, and the mixed frame.

[0044] It is understandable that the text line can be further divided into printed lines and handwritten lines. The printed lines can be understood as the question text lines, and the handwritten lines can be understood as the answer text lines. Each text line includes at least one character. The printed lines and the handwritten lines are detected to obtain a printed frame that selects the printed lines and a handwritten frame that selects the handwritten lines containing the answer text. It is understandable that the handwritten lines only include handwriting (answer text) and do not include printed text (question text). Correspondingly, the printed lines only include printed text and do not include handwriting. This separate detection method can accurately distinguish between questions and answers, which facilitates the subsequent construction of the target matrix. Subsequently, the overlap between the text boxes (IoU, Intersection over Union) is calculated or based on the position of each text box on the question image, the outer frames of the printed boxes and the handwritten boxes located in the same text line are merged to obtain a mixed box. The multiple text boxes include mixed boxes, unmerged printed boxes, and unmerged handwritten boxes.

[0045] Optionally, the above-mentioned determination of the first position of the target text box containing the answer text among the multiple text boxes in the question image, and matrixing the first position and the answer text to generate a target matrix can be specifically achieved through the following steps.

[0046] The position of a target text box containing the answer text among the multiple text boxes is determined as the first position in the question image, or the text boxes are sorted based on their positions in the question image, and the serial number of the target text box is determined as the first position; the row and column positions of the answer text are determined based on the first position and the second position of the answer text in the target text box; the row and column positions are associated with the answer text to generate a target matrix.

[0047] It is understandable that the position of the target text box in the question image is used as the first position. For example, the position of the target text box in the question image can be determined based on the pixels or feature points contained in the target text box. Alternatively, all text boxes can be sorted according to their positions in the question image, and the sequence number of the target text box can be used as the first position. Subsequently, the number of answer texts included in the target text box is determined, or the number of handwritten boxes included in the target text box is counted, and the second position of the answer text in the target text box is determined. The second position can also be the specific position of the answer text in the target text box, or the sequence number of the handwritten box in the target text box. In one possible embodiment, based on the first position of the target text box in the question image and the second position of the handwritten box in the target text box, the third position of the answer text selected by the handwritten box is determined. The third position can be understood as the row and column position in the target matrix. Subsequently, the third position is associated with the answer text to generate a target matrix. In the target matrix, the answer text and the corresponding target text box can be accurately determined based on the row and column positions, and then the question selected by the target text box can be locked, or the row and column positions can be directly associated with the target question selected by the target text box for quick positioning.

[0048] For example, see FIG2 , which is a schematic diagram of a title image provided by an embodiment of the present disclosure. Title image 1 is a detection schematic diagram obtained after single-row detection. Title image 1 includes multiple handwritten frames and multiple printed frames. The overlap of the printed frames and handwritten frames included in each row is calculated to obtain a mixed frame as shown in title image 2. For example, handwritten frame 11 and printed frame 12 are merged to obtain mixed frame 21, and handwritten frame 13, handwritten frame 14, and printed frame 15 are merged to obtain mixed frame 22. As shown in title image 2, the sequence number of mixed frame 21 in the title image is 1, that is, the first position is 1, and the first position can refer to the row position of the handwritten frame. The sequence number of handwritten frame 11 in mixed frame 21 is 1, that is, the second position is 1 (or represented as A), and the second position can be the column position of the handwritten frame. Then, based on the first position and the second position, the row and column position of the answer text in the handwritten body frame 11 can be determined to be 1A, which can be expressed as 1A=AA in the matrix, which can be understood as the answer content in the first row and first column is AA. Similarly, the matrix obtained based on the mixed body frame 22 is expressed as 2A=BB, 2B=CC, which can be understood as the answer content in the first row and first column is BB, and the answer content in the second row and second column is CC, that is, there are two handwritten answers in the second row, namely BB and CC. The sorting result in the target matrix can be expressed as (AA, BB-CC).

[0049] In step S102, the text recognition result and the question image are used as inputs of a pre-trained multimodal model to obtain an output result.

[0050] The output result includes the text to be corrected of the target question in the question image and the question type of the target question.

[0051] It can be understood that, based on the above step S101, the text recognition result and the question image are used as the input of a pre-trained multimodal model to obtain an output result. The multimodal model can use the MathGPT model as a basis. The multimodal model can process different modal data such as text, image and / or video. The multimodal model is used to perform question division and question classification. The output result of the multimodal model includes the text to be corrected of the target question in the question image and the question type of the target question. The target question refers to a question in the question image, or a question to be corrected selected by the user from the question image. The text to be corrected may include the answer text and the question text, or may only include the question text.

[0052] In step S103, a correction mode corresponding to the question type is determined, and a reference text is obtained based on the text to be corrected in the correction mode.

[0053] It is understood that, based on the above step S102, different question types correspond to different grading modes, and the multiple grading strategies set can be applied to different question types to expand the scope of problem solving and grading compatibility. The grading mode corresponding to the question type is determined, and a reference text is generated based on the text to be graded in this grading mode, or the grading results are directly generated in this grading mode, where the grading modes include a database search mode and a deductive problem solving mode.

[0054] In step S104, the reference text is compared with the answer text corresponding to the target question in the target matrix to generate a correction result for the target question.

[0055] It is understood that, based on the above steps S101 and S103, the answer text corresponding to the target question is determined in the target matrix. Based on the above example, if the target question is the first question, AA can be determined as the answer text for the target question. After obtaining the reference text, the reference text and the answer text are compared to generate a correction result for the target question. The correction result may include information such as whether the answer is correct, the correct answer, and the solution process.

[0056] The homework grading method provided in this application improves the efficiency of locating answers to questions by constructing a matrix composed of the order of answer texts; divides and classifies questions and extracts question information presented in different forms through a multimodal model, effectively improving the scope of problem solving; sets different problem-solving strategies based on different question types, and uses more suitable problem-solving strategies for grading, which can also improve the accuracy of grading to a certain extent; utilizes the wide adaptability and reasoning ability of machine learning models, and solves questions by performing logical reasoning on the grading text, which can understand similar or new question types, has stronger compatibility, and effectively improves the scope and accuracy of problem solving; and quickly and accurately feeds back the grading results to users, effectively improving the user experience.

[0057] FIG3 is a detailed flowchart of step S102 in the homework grading method provided in an embodiment of the present disclosure, wherein the text recognition result and the question image are used as inputs of a pre-trained multimodal model to obtain an output result, specifically including the following steps S301 and S302 as shown in FIG3 .

[0058] In step S301, the image area of ​​the target question in the question image is associated with the corresponding text area in the text recognition result to obtain the question data of the target question.

[0059] It can be understood that the image area of ​​the target question in the question image and the text area corresponding to the target question in the text recognition result are associated to obtain the question data of the target question. For graphic and text questions, the text recognition result includes the question text and the answer text, and the question image includes the graphics involved in the target question. In this case, by associating the image area and the text area, the graphic questions involved in the image area can be translated into text-type questions through the multimodal model to facilitate subsequent reasoning. This conversion method between modalities can correct a variety of question types and effectively increase the scope of problem solving.

[0060] In step S302, the topic type of the target topic is identified based on the topic data, and a recognition model corresponding to the topic type is called to generate a text to be corrected for the target topic based on the topic data.

[0061] It is understood that, based on the above step S301, the question data is identified to determine the question type of the target question. The question types include a first type and a second type. The first type can be understood as the data type of the question, such as pure text data, graphic data, and mixed text and graphic data, and the second type can be understood as the solution type of the question. After determining the question type, the second recognition model corresponding to the question type is called to generate the target question's uncorrected text based on the question data. The uncorrected text is used to subsequently generate the reference text.

[0062] The topic data further includes the hierarchical relationship between the target topic and the remaining topics in the topic image except the target topic.

[0063] It is understandable that the question data also includes the hierarchical relationship or level relationship between questions. The hierarchical relationship refers to the situation where a large question contains small questions. For example, an application question includes multiple small questions. The hierarchical relationship can be obtained by dividing the questions through a multimodal model. Specifically, the hierarchical relationship includes the correlation relationship between the target question and the remaining questions.

[0064] Optionally, in the case where the various topics included in the topic image have a hierarchical relationship, the above-mentioned calling of the recognition model corresponding to the topic type generates the text to be corrected of the target topic based on the topic data, which can be specifically achieved through the following steps.

[0065] Call the recognition model corresponding to the question type to generate an initial text based on the question data; obtain the filler text corresponding to the target question from the remaining questions based on the hierarchical relationship, wherein the filler text refers to the shared information between all questions at the same level; generate the text to be corrected of the target question based on the initial text and the filler text.

[0066] It is understandable that the multimodal model also includes multiple second recognition models, which are used to recognize texts of different question types. The second recognition model corresponding to the question type is called to perform text and / or graphic recognition based on the question data to generate initial text.

[0067] Understandably, one possible scenario is to obtain the fill-in text corresponding to the target question from the remaining questions based on the hierarchical relationship. The fill-in text refers to the shared information between all questions at the same level as the target question and at that level. For example, the target question is the third question in the main question, and the text involved in the third question is related to a text in the second question. Then, a text in the second question is used as the fill-in text for the third question, so as to obtain the complete text to be corrected for the target question based on the initial text and the fill-in text, so as to facilitate the subsequent generation of the reference text directly based on the text to be corrected, that is, to obtain the fill-in text from the remaining questions before correction.

[0068] Optionally, the second recognition model corresponding to the question type is called to generate an initial text based on the question data, and the hierarchical relationship and the initial text are associated to obtain the text to be revised, so as to obtain a reference text based on the text to be revised in the revision mode.

[0069] Understandably, another possible scenario is that after obtaining the initial text, the hierarchical relationship is associated with the initial text to obtain the target question's text to be corrected, or the initial text of the target question is associated with the initial text of the remaining questions based on the hierarchical relationship to obtain the target question's text to be corrected. Subsequently, when correcting the text to be corrected, the reference text of the target question is obtained based on the initial text of the target question and the initial text of the remaining questions, that is, the filler text is obtained from the remaining questions during the correction process.

[0070] It is understandable that, when there is no hierarchical relationship between the various topics included in the topic image, the second recognition model corresponding to the topic type is called to generate the text to be corrected based on the topic data.

[0071] The question type includes the first type.

[0072] It is understandable that the topic type includes a first type, which refers to the data modality included in the topic information of the target topic, for example, the topic information includes both text and graphics, or the topic information only includes text or graphics.

[0073] Optionally, the above-mentioned calling of the recognition model corresponding to the question type generates the initial text based on the question data, or calling of the recognition model corresponding to the question type generates the text to be corrected based on the question data, which can be specifically achieved through the following steps.

[0074] In the case where the first type is a text question, a pre-trained text recognition model is called to generate the initial text of the target question based on the question data; or, in the case where the first type is a graphic question, a pre-trained graphic recognition model is called to generate the initial text based on the question data; or, in the case where the first type is a graphic-text question, the initial text is obtained by calling the graphic recognition model to generate the first text based on the image area and calling the text recognition model to generate the second text based on the text area; or, calling the pre-trained graphic-text recognition model to generate the initial text based on the question data.

[0075] It is understandable that in the case where the first type is a text question, that is, the question information is presented in the form of text, such as fill-in-the-blank questions and true-or-false questions, in this case, calling a pre-trained text recognition model to generate the target question's text to be corrected based on the question data can be understood as performing text recognition again, and the text to be corrected is the text recognition result. The text area in the text recognition result can also be directly used as the text to be corrected. In the case where the first type is a graphic question, that is, the question information is presented in the form of a graphic, such as a connection question and a box selection question, in this case, calling a graphic recognition model to perform recognition based on the image area in the question information, and combining the graphic recognition result with the recognition result included in the text area to generate the initial text. In the case where the first type is a graphic-text question, that is, the question information is presented through text and graphics, such as a geometry question, in this case, calling a graphic-text recognition model to generate the initial text based on the question data, or calling a graphic recognition model to generate the first text based on the image area, calling a text recognition model to generate the second text based on the text area, and then generating the initial text based on the first text and the second text.

[0076] The homework grading method provided in this application takes the question image and text recognition results as the input of the multimodal model, and outputs the question type, the text to be graded, and the hierarchy of the sub-questions contained in the big question through the multimodal model. The descriptive text of the graphic question can be extracted to generate the text to be graded, which is convenient for subsequent grading of different question types, effectively improving the scope of problem solving, and eliminating the need to detect the model of the big question and compare the answers to obtain the question type, thereby improving the grading efficiency to a certain extent.

[0077] FIG4 is a detailed flowchart of step S103 in the homework grading method provided by an embodiment of the present disclosure, wherein determining the grading mode corresponding to the question type and obtaining a reference text based on the text to be graded under the grading mode specifically includes the following steps S401 and S402 as shown in FIG4 .

[0078] The question type includes the second type.

[0079] It is understandable that the question type also includes the second type, which refers to the problem-solving method. The second type includes calculation type and reasoning type. For example, when solving a problem, multiple reasonings are required to obtain a reference answer, or only calculations are required to obtain a reference answer, and calculations may also be involved in the reasoning process.

[0080] In step S401, when the second type is a calculation type, a calculation plug-in is called to perform calculation based on the text to be corrected to obtain the reference text.

[0081] It is understandable that when the second type is a calculation type, such as oral arithmetic questions and formula questions, in this case, the calculation plug-in is called to perform calculations based on the text to be corrected to obtain a reference text. That is, the plug-in can be called to call a mature oral arithmetic correction service for all oral arithmetic questions and obtain an oral arithmetic correction result. The oral arithmetic correction result can be a reference text or a correction result of whether it is correct or not. Among them, the oral arithmetic correction service supports corrections such as horizontal and vertical forms, escaped forms, equation solving, and unit conversion.

[0082] In step S402, when the second type is an inference type, a question sample corresponding to the text to be corrected is searched in a pre-constructed database, and the reference text is obtained based on the answer to the question sample; or, when the question sample is not found, a pre-trained correction model is called based on the text to be corrected to obtain the reference text.

[0083] It is understandable that when the second type is a reasoning type, such as application, selection, fill-in-the-blank, judgment questions, etc., in this case, the question bank search plug-in is called to search the database for question samples corresponding to the text to be corrected. The question samples can be exactly the same or partially the same as the text to be corrected. Specifically, the similarity between the question samples and the text to be corrected can be calculated, and the reference answer can be further obtained based on the answer to the question sample. When the question sample is not found in the database, or when it is determined that the second type is a reasoning type, a pre-trained correction model is called to correct the text to be corrected and output a reference text. At least one correction method of question bank search and model correction can be performed.

[0084] In one possible embodiment, the target question is a multiple-choice, fill-in-the-blank, or judgment question, and a question bank search plug-in is called. The plug-in can be an application programming interface (API). If the database has a question sample and a correct answer (reference answer) that are identical to the text to be corrected, the reference answer can be compared with the answer text in the target matrix and the correction result can be returned. If no question sample is found, the correction model for multiple-choice, fill-in-the-blank, and judgment in MathGPT is called to correct the text to be corrected, output the reference answer, and compare it with the answer text to obtain a correction result of whether it is correct or not. At the same time, the information such as the text to be corrected, the question image, and the reference answer are associated and stored in the question bank.

[0085] The reasoning type includes application questions.

[0086] Optionally, the above-mentioned comparison of the reference text with the answer text corresponding to the target question in the target matrix to generate the correction result of the target question can be specifically achieved through the following steps.

[0087] The reference text and the answer text corresponding to the target question are semantically analyzed using a pre-trained application question model to determine whether the answer text and the reference text have the same semantics, and output a judgment result; the correction result of the target question is generated based on the judgment result.

[0088] It is understandable that when the target question is of the reasoning type and is an application question in the reasoning type, the question bank search plug-in is called. If there are question samples and correct answers (standard answers / answer texts) corresponding to the text to be corrected in the question bank, the application question model is used to determine whether the answer text and the reference answer in the target matrix have the same semantics, that is, to determine whether the handwritten answer and the standard answer have the same meaning. For example, the standard answer is "Xiaohong has a total of 5 apples", and the handwritten answer is "The one with 5 apples is Xiaohong". Through semantic analysis by the application question model, the judgment result that the handwritten answer and the standard answer have the same semantics can be output, and the correction result is generated based on the judgment result. The correction result includes whether the target question is written correctly or incorrectly.

[0089] Understandably, for word problems, if no question sample is found in the question bank, the word problem solving plug-in is used to analyze step by step through the thinking chain. When calculating numbers, the calculation plug-in can be called to obtain the calculation results, and finally the next step of analysis is carried out until the final result is obtained (reference text).

[0090] It is understandable that for connection questions, picture-based topic questions, geometry questions, etc., the graphics in the questions are converted into text through the recognition model to obtain descriptive text. For example, connection questions are converted into a connecting b, c connecting d, etc., geometry questions have descriptive text about auxiliary lines, and picture-based topic questions have descriptive text about graphics. The recognition model generates the text to be corrected based on the descriptive text and the recognized text in the text recognition results.

[0091] The homework grading method provided in this application provides multiple grading strategies. By combining the basic grading strategy and the machine learning model grading strategy, when the same question cannot be found in the question bank, the model grading method can also be used to ensure that the homework grading can be completed. In addition, the grading accuracy can be improved by verifying the grading results of multiple correction strategies.

[0092] On the basis of the above embodiment, FIG5 is a flow chart of another homework grading method provided by the embodiment of the present disclosure, which is applied to the scenario where the user takes a picture of homework through the front camera of the learning machine, and specifically includes the following processes: (1) obtaining the photographed question image and performing document correction on the question image; (2) detecting the printed text lines and handwritten text lines in the corrected question image, and merging the outer frames of the printed text frame and the handwritten text frame located in the same text line through the line forming algorithm, and sorting them from top to bottom according to the order of the handwritten text lines to obtain a target matrix; (3) recognizing each text line in the question image through the text recognition model to obtain the text recognition result of the whole page; (4) taking the text recognition result and the question image as the input of the multimodal model, splitting the question through the multimodal model, separating the text recognition result according to the large question to determine the hierarchical relationship between the questions, and marking the question type of each question to obtain the text to be graded for each question; (5) calling different grading models according to the question type for classification and correction, and executing (6)-( 8) The corresponding steps are used to obtain the correction results; (6) If the question type is oral arithmetic, the calculation plug-in is called and the correction results are returned; (7) If the question type is multiple choice, fill-in-the-blank, true or false, connection question, geometry question and picture-based topic, the question bank search plug-in is called. If the same question and correct answer exist in the question bank, the correct answer is compared with the handwritten answer in the target matrix and the correction results are returned; or, the correction model is called to correct the text to be corrected and the correction results are returned; (8) If the question type is an application question, the question bank search plug-in is called. If the same question and correct answer exist in the question bank, the application question model is used to determine whether the correct answer and the handwritten answer have the same meaning and the correction results are returned; or, the application question solving plug-in is used to perform step-by-step analysis through the thinking chain. When calculation is involved, the calculation plug-in is called to obtain the calculation result for subsequent analysis, and the application result is compared with the handwritten answer and the correction results are returned; (9) The relevant information of the question is written into the question bank.

[0093] It is understandable that the specific implementation instructions involved in the above-mentioned homework correction method can be found in the above-mentioned embodiment, which will not be repeated here.

[0094] Figure 6 is a structural diagram of a homework correction device provided by an embodiment of the present disclosure. The homework correction device is applied to the server side. The device 600 is used to execute the above-mentioned homework correction method. The device 600 specifically includes an acquisition unit 601, an identification unit 602, a correction unit 603 and a generation unit 604.

[0095] The acquisition unit 601 is used to acquire a question image, and perform text recognition on the question image to generate a text recognition result and a target matrix, wherein the target matrix is ​​composed of the order of the answer texts included in the question image.

[0096] The recognition unit 602 is used to use the text recognition result and the question image as inputs of a pre-trained multimodal model to obtain an output result, which includes the text to be corrected of the target question in the question image and the question type of the target question.

[0097] The correction unit 603 is used to determine a correction mode corresponding to the question type, and obtain a reference text based on the text to be corrected under the correction mode.

[0098] The generating unit 604 is configured to compare the reference text with the answer text corresponding to the target question in the target matrix to generate a correction result for the target question.

[0099] Optionally, the acquisition unit 601 is used to: detect text lines in the question image to obtain multiple text boxes that select each text line; determine the first position of the target text box containing the answer text among the multiple text boxes in the question image, and matrix the first position and the answer text to generate a target matrix; perform text recognition on the multiple text boxes to generate a text recognition result.

[0100] Optionally, the text lines in the device 600 include printed lines and handwritten lines.

[0101] Optionally, the acquisition unit 601 is used to: detect the printed line and the handwritten line in the title image, obtain a printed frame that selects the printed line and a handwritten frame that selects the handwritten line; merge the printed frame and the handwritten frame located in the same text line to obtain a mixed frame; wherein the multiple text frames include the printed frame and the mixed frame that have not been merged.

[0102] Optionally, the acquisition unit 601 is used to: determine the position of the target text box containing the answer text among the multiple text boxes in the question image as the first position, or, sort based on the position of each text box in the question image, and determine the serial number of the target text box as the first position; determine the row and column positions of the answer text based on the first position and the second position of the answer text in the target text box; associate the row and column positions with the answer text to generate a target matrix.

[0103] Optionally, the recognition unit 602 is used to: associate the image area of ​​the target question in the question image with the corresponding text area in the text recognition result to obtain the question data of the target question; identify the question type of the target question based on the question data, and call the recognition model corresponding to the question type to generate the text to be corrected of the target question based on the question data.

[0104] Optionally, the topic data in the apparatus 600 further includes a hierarchical relationship between the target topic and other topics in the topic image except the target topic.

[0105] Optionally, the recognition unit 602 is used to: call the recognition model corresponding to the question type to generate an initial text based on the question data; obtain the filler text corresponding to the target question from the remaining questions based on the hierarchical relationship, wherein the filler text refers to shared information between all questions at the same level; and generate the text to be corrected for the target question based on the initial text and the filler text.

[0106] Optionally, the topic type in the device 600 includes the first type.

[0107] Optionally, the recognition unit 602 is used to: when the first type is a text question, call a pre-trained text recognition model to generate the initial text of the target question based on the question data; or, when the first type is a graphic question, call a pre-trained graphic recognition model to generate the initial text based on the question data; or, when the first type is a graphic-text question, obtain the initial text based on the first text generated by calling the graphic recognition model based on the image area and the second text generated by calling the text recognition model based on the text area; or, call a pre-trained graphic-text recognition model to generate the initial text based on the question data.

[0108] Optionally, the question type in the device 600 includes the second type.

[0109] Optionally, the correction unit 603 is used to: when the second type is a calculation type, call a calculation plug-in to perform calculations based on the text to be corrected to obtain the reference text; when the second type is an inference type, search a pre-constructed database for question samples corresponding to the text to be corrected, and obtain the reference text based on the answers to the question samples; or, call a pre-trained correction model to obtain the reference text based on the text to be corrected.

[0110] Optionally, the reasoning type in the device 600 includes application questions.

[0111] Optionally, the generation unit 604 is used to: perform semantic analysis on the reference text and the answer text corresponding to the target question through a pre-trained application question model to determine whether the answer text and the reference text have the same semantics, and output the judgment result; generate the correction result of the target question based on the judgment result.

[0112] The device provided in this embodiment has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0113] The exemplary embodiments of the present disclosure further provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being configured to cause the electronic device to perform a method according to an exemplary embodiment of the present disclosure when executed by the at least one processor.

[0114] Exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to cause the computer to perform the method according to the embodiment of the present disclosure.

[0115] With reference to Figure 7, a block diagram of an electronic device 700 that can serve as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0116] As shown in Figure 7, electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In RAM 703, various programs and data required for the operation of device 700 can also be stored. Computing unit 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0117] Multiple components within electronic device 700 are connected to I / O interface 705, including an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. Input unit 706 can be any type of device capable of inputting information into electronic device 700. Input unit 706 can receive input numeric or character information and generate key input signals related to user settings and / or function control of the electronic device. Output unit 707 can be any type of device capable of presenting information and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 708 may include, but is not limited to, a magnetic disk or an optical disk. Communication unit 709 allows electronic device 700 to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0118] The computing unit 701 can be various general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the homework grading method or the training method of the machine learning model for multiple auxiliary grading can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. In some embodiments, the computing unit 701 can be configured to execute the homework grading method or the training method of the machine learning model for multiple auxiliary grading by any other appropriate means (e.g., by means of firmware).

[0119] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0120] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0121] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., as an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0124] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact via a communication network. The client and the server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other.

[0125] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for marking homework, comprising: Acquire a question image, and perform text recognition on the question image to generate a text recognition result and a target matrix, wherein the target matrix is ​​generated based on the answer text included in the question image; Using the text recognition result and the question image as inputs of a pre-trained multimodal model to obtain an output result, wherein the output result includes the text to be corrected of the target question in the question image and the question type of the target question; Determining a correction mode corresponding to the question type, and obtaining a reference text based on the text to be corrected in the correction mode; The reference text is compared with the answer text corresponding to the target question in the target matrix to generate a correction result for the target question.

2. The method according to claim 1, characterized in that The performing text recognition on the title image to generate a text recognition result and a target matrix includes: Detecting text lines in the title image and obtaining a plurality of text boxes for selecting each text line; Determine a first position of a target text box containing answer text in the plurality of text boxes in the question image, and perform a matrix operation on the first position and the answer text to generate a target matrix; Perform text recognition on the multiple text boxes to generate text recognition results.

3. The method according to claim 2, characterized in that Each of the text lines includes a printed line and / or a handwritten line. The detecting of the text lines in the title image to obtain a plurality of text boxes for selecting each text line includes: Detecting the printed lines and the handwritten lines in the question image, and obtaining a printed frame that selects the printed lines and a handwritten frame that selects the handwritten lines; Merging the printed body frame and the handwritten body frame located in the same text line to obtain a mixed body frame; The multiple text frames include the unmerged printed frames, the unmerged handwritten frames, and the mixed frames.

4. The method according to claim 2 or 3, characterized in that The step of determining a first position of a target text box containing answer text in the plurality of text boxes in the question image, and performing a matrix operation on the first position and the answer text to generate a target matrix includes: Determine the position of a target text box containing the answer text among the multiple text boxes in the question image as a first position, or, sort the text boxes based on their positions in the question image, and determine the sequence number of the target text box as the first position; Determining the row and column position of the answer text based on the first position and the second position of the answer text in the target text box; The row and column positions are associated with the answer text to generate a target matrix.

5. The method according to any one of claims 1 to 4, characterized in that The text recognition result and the title image are used as inputs of a pre-trained multimodal model to obtain an output result, including: Associating the image area of ​​the target question in the question image with the corresponding text area in the text recognition result to obtain question data of the target question; The topic type of the target topic is identified based on the topic data, and a recognition model corresponding to the topic type is called to generate a text to be corrected for the target topic based on the topic data.

6. The method according to claim 5, characterized in that The question data also includes a hierarchical relationship between the target question and the remaining questions in the question image except the target question. The calling of the recognition model corresponding to the question type and generating a text to be corrected for the target question based on the question data includes: Calling the recognition model corresponding to the question type and generating the initial text of the target question based on the question data; Obtaining filler text corresponding to the target topic from the remaining topics based on the hierarchical relationship, wherein the filler text refers to shared information between all topics at the same level; The text to be corrected of the target topic is generated according to the initial text and the fill-in text.

7. The method according to claim 5, wherein: The question data also includes a hierarchical relationship between the target question and the remaining questions in the question image except the target question. The calling of the recognition model corresponding to the question type and generating a text to be corrected for the target question based on the question data includes: Calling the recognition model corresponding to the question type and generating the initial text of the target question based on the question data; The hierarchical relationship is associated with the initial text, or the initial text is associated with the remaining topics based on the hierarchical relationship to generate a text to be corrected for the target topic.

8. The method according to claim 7, wherein: The determining of the correction mode corresponding to the question type and obtaining a reference text based on the text to be corrected in the correction mode includes: Obtaining filler text corresponding to the target topic from the remaining topics based on the hierarchical relationship, wherein the filler text refers to shared information among all topics at the same level; and The text to be corrected of the target topic is generated according to the text to be corrected and the filler text.

9. The method according to any one of claims 5 to 8, characterized in that The question type includes a first type, and calling the recognition model corresponding to the question type to generate a text to be corrected for the target question based on the question data includes: In the case where the first type is a text question, a pre-trained text recognition model is called to generate the initial text of the target question based on the question data; or In the case where the first type is a graphic question, calling a pre-trained graphic recognition model to generate the initial text of the target question based on the question data; or In the case where the first type is a picture-text question, the initial text of the target question is obtained by calling the first text generated by the graphic recognition model based on the image area and calling the second text generated by the text recognition model based on the text area; or, calling a pre-trained picture-text recognition model to generate the initial text of the target question based on the question data.

10. The method according to claim 9, wherein: The first type is the graphic question, and the calling of a pre-trained graphic recognition model to generate the initial text of the target question based on the question data includes: Converting the graphics in the target topic into text using the graphics recognition model to obtain a description text; and The initial text of the target topic is generated based on the description text and the recognized text in the text recognition result.

11. The method according to any one of claims 1 to 10, characterized in that The question type further includes a second type, and determining a correction mode corresponding to the question type and obtaining a reference text based on the text to be corrected in the correction mode includes: In the case where the second type is a calculation type, calling a calculation plug-in to perform calculation based on the text to be corrected to obtain the reference text; or In the case where the second type is the reasoning type, a question sample corresponding to the text to be corrected is searched in a pre-built database, and the reference text is obtained based on the answer to the question sample; or, a pre-trained correction model is called to obtain the reference text based on the text to be corrected.

12. The method according to claim 11, characterized in that The reasoning type includes application questions, and the comparison of the reference text with the answer text corresponding to the target question in the target matrix to generate the correction result of the target question includes: Performing semantic analysis on the reference text and the answer text corresponding to the target question using a pre-trained application question model to determine whether the answer text and the reference text have the same semantics, and outputting a determination result; A correction result of the target question is generated according to the judgment result.

13. The method according to claim 11 or 12, wherein: In the case where the second type is the reasoning type, the method further includes: associating the text to be corrected, the question image and the reference text and storing them in the database.

14. A homework grading device, comprising: an acquisition unit, configured to acquire a question image, perform text recognition on the question image, and generate a text recognition result and a target matrix, wherein the target matrix is ​​formed based on the answer text included in the question image; a recognition unit, configured to use the text recognition result and the question image as inputs to a pre-trained multimodal model to obtain an output result, wherein the output result includes the text to be corrected of the target question in the question image and the question type of the target question; a correction unit, configured to determine a correction mode corresponding to the question type, and obtain a reference text based on the text to be corrected under the correction mode; The generating unit is used to compare the reference text with the answer text corresponding to the target question in the target matrix to generate a correction result of the target question.

15. A computer program product, wherein The computer program product comprises a computer program which, when executed by a processor of a computer, causes the computer to perform the method according to any one of claims 1 to 13 .

16. An electronic device, wherein: The electronic device comprises: at least one processor; and at least one storage device coupled to the at least one processor; The at least one processor is configured to read the computer program stored in the at least one memory to execute the method according to any one of claims 1 to 13.

17. A non-transitory computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, which, when executed by a processor of a computer, causes the computer to perform the method according to any one of claims 1 to 13.

Images