Method and device for framing questions in student homework scenarios, terminal equipment and storage medium

By detecting and integrating the number, area and line area information of the question, combining text recognition and semantic classification, the problem of poor box selection in the existing technology is solved, and the accurate box selection and tilting or arching question areas in students' homework is achieved. 多级题目序号处理,提升了框选的准确性和效率。

CN112270295BActive Publication Date: 2024-11-26GUANGDONG XIAOTIANCAI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011267013.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-11-13
Publication Date
2024-11-26
Estimated Expiration
2040-11-13

AI Technical Summary

Technical Problem

The existing framed question method is difficult to accurately identify and framed in student homework scenarios and framed inclined or arched question areas, and cannot effectively deal with scenes containing multiple question numbers at different levels, resulting in poor box selection results.

Method used

By obtaining the target picture, detecting and integrating the question number, question area and line area information, determine the final question area range, and perform text recognition and semantic classification, and output box selection range marks to guide the box selection process.

Benefits of technology

It improves the accuracy and efficiency of box selection, ensures complete content, and is suitable for tilted or arched question areas, avoids wrong boxes and missing boxes, and is especially suitable for complex scenarios such as test papers and exercise books.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112270295B_ABST
    Figure CN112270295B_ABST
Patent Text Reader

Abstract

The present invention provides a method and device for framing questions in a student homework scenario, a terminal device and a storage medium, wherein the method for framing questions in a student homework scenario includes: obtaining a target image containing questions to be framed; detecting the question numbers of each level in the target image, the question areas corresponding to each question number and the question row areas respectively; fusing the detected question numbers, the question areas corresponding to each question number and the question row area information to determine the final question area range corresponding to each question number; performing text recognition and semantic classification on the question area corresponding to the highest level question number to obtain the frame selection range mark of each question; and framing each question in the image according to the frame selection range mark. The method for framing questions is particularly suitable for situations where test papers, exercise books, books, etc. are tilted or arched in the target image, ensuring that the frame selection result is complete, improving the efficiency of framing questions, and playing a good supporting role in subsequent business.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for framing questions in a student homework scenario, a terminal device and a storage medium. Background Art

[0002] At present, apps that help students do homework or correct their homework are becoming more and more popular. In order to better meet business needs, how to accurately locate the question area contained in the pictures of test papers, exercise books, books, etc. has become a key link in the entire technical chain.

[0003] Due to the ever-changing learning scenes of students, the ambient light, the relative position of the camera and the paper, the degree of convexity of the paper, etc. will all change to a certain extent during shooting, which poses a great challenge to solving the above problems. There are generally two major problems when current solution providers deal with it: 1) The framed question area is rectangular, which has a poor effect when the shooting target is tilted or arched, and it is easy to have multiple or fewer frames; 2) The same complete question may contain multiple question areas with different levels of question numbers, resulting in poor question selection results. Summary of the invention

[0004] The purpose of the present invention is to provide a method and apparatus for framing questions in a student homework scenario, a terminal device and a storage medium, which effectively solve the technical problem that the existing framing method has poor framing effect when the questions in the target image are tilted / arched, or when the target image contains question areas with multiple question numbers of different levels.

[0005] The technical solution provided by the present invention is as follows:

[0006] The present invention provides a method for setting a question frame in a student homework scenario, comprising:

[0007] Get the target image containing the title to be selected;

[0008] Detecting the question numbers at each level in the target image, the question areas corresponding to the question numbers, and the question row areas respectively;

[0009] The detected question numbers, the question areas corresponding to the question numbers, and the question row area information are integrated to determine the final question area range corresponding to the question numbers;

[0010] Perform text recognition and semantic classification on the question area corresponding to the highest level question number to obtain the selection range mark of each question;

[0011] Each title in the picture is framed and selected according to the frame selection range mark.

[0012] In this technical solution, the question numbers in the target image, the question areas corresponding to each question number, and the question row areas are detected respectively, and the final question area range corresponding to each question number is obtained by fusion, so as to avoid the accuracy of area detection affected by wrong frames and missed frames during the question area detection process, and improve the efficiency of framing questions. In addition, after determining the final question area range corresponding to each question number, the question area corresponding to the highest level question number is subjected to text recognition and semantic classification to determine whether it is a large question or a small question, so as to avoid framing errors. This framing question method is particularly suitable for situations where test papers, exercise books, books, etc. are tilted or arched in the target image, so as to ensure that the results of the framing selection are complete and provide good support for subsequent business.

[0013] Further preferably, the detecting of the question numbers at each level in the target image, the question areas corresponding to the question numbers, and the question row areas respectively comprises executing the following steps in parallel:

[0014] Locate and identify the serial numbers of each level of questions in the target image to obtain the serial numbers of each level of questions;

[0015] The question areas corresponding to the question numbers at each level are treated as a whole and divided along the edge of the question to obtain the question areas corresponding to the question numbers;

[0016] Split each line of text in the title as a whole to obtain the title line area.

[0017] Further preferably, the detected question numbers, the question areas corresponding to the question numbers, and the question row area information are integrated to determine the final question area range corresponding to the question numbers, including:

[0018] Matching the topic serial number and topic row area information with the topic area information;

[0019] When mismatch information appears, the topic area is adjusted according to the topic serial number and / or topic row area information to determine the final topic area range corresponding to each topic serial number.

[0020] In this technical solution, the feature extraction capability of the AI ​​algorithm is fully utilized to extract the question number, the question area corresponding to each question number, and the question row area information in turn, and then give full play to the role of information fusion, so that the entire framing question algorithm can achieve excellent framing question effects.

[0021] Further preferably, the text recognition and semantic classification are performed on the question area corresponding to the highest level question number to obtain the selection range mark of each question, including:

[0022] Send the question area corresponding to the highest level question number to a text recognition tool for text recognition;

[0023] Semantically classify the recognized text and calculate the correlation between the semantics based on the classification results;

[0024] comparing the calculated semantic relevance with a preset relevance threshold;

[0025] Output the selection range mark of the question according to the comparison result.

[0026] In this technical solution, by performing text recognition and semantic classification on the question area corresponding to the highest level question number, the corresponding selection range mark is output, and the question area is subsequently selected according to the selection range mark, which greatly avoids the occurrence of incorrect selection. In particular, when processing questions containing multiple levels of question numbers, the required question area can be correctly selected.

[0027] Further preferably, the frame selection range mark includes: a mark for frame selection including the question area corresponding to the highest level question number and a mark for frame selection not including the question area corresponding to the highest level question number;

[0028] When the calculated semantic relevance is less than the preset relevance threshold, the output box selection does not include the mark of the question area corresponding to the highest level question number;

[0029] When the calculated semantic relevance is greater than or equal to the preset relevance threshold, the output box selects the mark of the question area corresponding to the highest level question number.

[0030] The present invention also provides a frame question device in a student homework scenario, comprising:

[0031] An image acquisition module is used to acquire a target image containing the title to be framed;

[0032] A detection module, used to detect the question numbers of each level, the question areas corresponding to the question numbers, and the question row areas in the target image acquired by the image acquisition module;

[0033] An information fusion module is used to fuse the question numbers, question areas corresponding to each question number, and question row area information detected by the detection module to determine the final question area range corresponding to each question number;

[0034] A semantic analysis module, used to perform text recognition and semantic classification on the question area corresponding to the highest level question number determined by the information fusion module, and obtain a selection range mark of each question;

[0035] The title selection module is used to select each title in the picture according to the selection range mark output by the semantic analysis module.

[0036] In this technical solution, the question numbers in the target image, the question areas corresponding to each question number, and the question row areas are detected respectively, and the final question area range corresponding to each question number is obtained by fusion, so as to avoid the accuracy of area detection affected by wrong frames and missed frames during the question area detection process, and improve the efficiency of framing questions. In addition, after determining the final question area range corresponding to each question number, the question area corresponding to the highest level question number is subjected to text recognition and semantic classification to determine whether it is a large question or a small question, so as to avoid framing errors. This framing question method is particularly suitable for situations where test papers, exercise books, books, etc. are tilted or arched in the target image, so as to ensure that the results of the framing selection are complete and provide good support for subsequent business.

[0037] Further preferably, the detection module includes:

[0038] The question number detection unit is used to locate and identify the question numbers of each level in the target image to obtain the question numbers of each level;

[0039] A question area detection unit is used to divide the question areas corresponding to the question numbers at each level as a whole and along the question edges to obtain the question areas corresponding to the question numbers;

[0040] The title line detection unit is used to segment each line of text in the title as a whole to obtain the title line area.

[0041] In this technical solution, the feature extraction capability of the AI ​​algorithm is fully utilized to extract the question number, the question area corresponding to each question number, and the question row area information in turn, and then give full play to the role of information fusion, so that the entire framing question algorithm can achieve excellent framing question effects.

[0042] Further preferably, the information fusion module includes:

[0043] An information matching unit, used for matching the topic sequence number and topic row area information with the topic area information;

[0044] The area adjustment unit is used to adjust the topic area according to the topic serial number and / or topic row area information when mismatch information appears, and determine the final topic area range corresponding to each topic serial number.

[0045] Further preferably, the semantic analysis module includes:

[0046] A character recognition unit, used for sending the question area corresponding to the highest level question number to a character recognition tool for character recognition;

[0047] A semantic classification unit, used to semantically classify the characters recognized by the character recognition unit, and calculate the correlation between the semantics according to the classification results;

[0048] A comparison unit, used for comparing the semantic relevance calculated by the semantic classification unit with a preset relevance threshold;

[0049] The result output unit is used to output the selection range mark of the question according to the comparison result of the comparison unit.

[0050] In this technical solution, by performing text recognition and semantic classification on the question area corresponding to the highest level question number, the corresponding selection range mark is output, and the question area is subsequently selected according to the selection range mark, which greatly avoids the occurrence of incorrect selection. In particular, when processing questions containing multiple levels of question numbers, the required question area can be correctly selected.

[0051] Further preferably, the frame selection range mark includes: a mark for frame selection including the question area corresponding to the highest level question number and a mark for frame selection not including the question area corresponding to the highest level question number;

[0052] When the comparison unit determines that the semantic relevance is less than a preset relevance threshold, the result output unit outputs a mark that the box selection does not include the question area corresponding to the highest level question number;

[0053] When the comparison unit determines that the semantic relevance is greater than or equal to the preset relevance threshold, the result output unit outputs a mark for selecting the question area corresponding to the highest level question number.

[0054] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the framing method in the above-mentioned student homework scenario when executing the computer program.

[0055] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the framing method in the above-mentioned student homework scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The preferred embodiments will be described below in a clear and understandable manner with reference to the accompanying drawings to further illustrate the above characteristics, technical features, advantages and their implementation methods.

[0057] Figure 1 It is a flow chart of an embodiment of a frame question method in a student homework scenario of the present invention;

[0058] Figure 2 A diagram showing the result of detecting the question area in an example of the present invention;

[0059] Figure 3 A diagram showing the test result of the question number in an example of the present invention;

[0060] Figure 4 A diagram showing a test result of a question row in an example of the present invention;

[0061] Figure 5 A schematic diagram of a topic area fed into a text recognition tool in an embodiment of the present invention;

[0062] Figure 6 A schematic diagram of selecting each title in a picture according to a selection range mark in an example of the present invention;

[0063] Figure 7 It is a structural schematic diagram of an embodiment of a frame question device in a student homework scenario of the present invention;

[0064] Figure 8 It is a schematic diagram of the terminal device structure in the present invention.

[0065] Description of Figure Numbers:

[0066] The title frame device 100, 110-image acquisition module, 120-detection module, 130-information fusion module, 140-semantic analysis module, 150-title frame selection module. DETAILED DESCRIPTION

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific embodiments of the present invention will be described below with reference to the accompanying drawings. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings and other embodiments can be obtained based on these drawings without creative work.

[0068] The first embodiment of the present invention, as Figure 1 As shown, a method for framing questions in a student homework scenario includes:

[0069] S10 obtains a target image containing a topic to be framed;

[0070] S20 detects the question numbers at each level in the target image, the question areas corresponding to the question numbers, and the question row areas respectively;

[0071] S30: The detected question numbers, the question areas corresponding to the question numbers, and the question row area information are integrated to determine the final question area range corresponding to the question numbers;

[0072] S40 performs text recognition and semantic classification on the question area corresponding to the highest level question number to obtain a selection range mark of each question;

[0073] S50 selects each title in the picture according to the selection range mark.

[0074] In this embodiment, the frame question method in the student homework scene is applicable to smart devices such as tutoring machines, learning machines, mobile phones or tablet computers with learning functions. The target image can be obtained by user input or by shooting with a camera configured in the smart device, which can contain at least one question. When it is shot by a camera configured in the smart device, the corresponding APP in the smart device is started to enter the shooting scene or the corresponding APP is started by communicating with an external electronic device. The target object is a book, exercise book, homework book, test paper, newspaper and other learning materials, and the camera device obtains the target image by shooting the target object. When shooting, there are no strict restrictions on the ambient light, the relative position between the camera and the paper, and the degree of concave and convex of the paper, and the shooting can be free. The opening of the shooting scene in the smart device can be set according to the actual situation, such as when the user uses books, exercise books, homework books, test papers and other learning materials, double-clicking the paper with fingers, stylus pens, pencils, rulers, sticks, etc. to trigger. For the picture input for detection in step S20, part of the image in the captured picture can also be selected manually or according to certain rules, which can be set freely, as long as the target picture for detection in step S20 contains at least one question.

[0075] Since there are not too many restrictions on the conditions for shooting the target image, it is difficult to extract the feature information of the target image using traditional methods. In order to solve this problem, in this embodiment, different models are created to detect the question numbers at all levels in the target image, the question areas corresponding to the question numbers, and the question row areas. The question areas corresponding to the question numbers at all levels are finally determined by combining the three aspects of information through fusion, so as to avoid wrong frames and missing frames in the question area detection process.

[0076] Generally speaking, each question in the learning material has a corresponding question number, and each question number may have a question number of the next level, and there may even be 3rd and 4th level numbers. For example, 4 categories of question number identification forms can be adopted, including: Serial number category 1: one, two, three, ... ten, eleven, ...; Serial number category 2: 1, 2, 3, ... 10, 11, ...; Serial number category 3: (1), (2), (3), ... (10), (11), ...; Serial number category 4: examples, additional questions. Here we do not make specific restrictions on the category form of the question number, and can adjust it according to the actual application situation. For example, in other embodiments, I, II, III, ... can also be added as other serial number categories. However, it should be noted that among the three serial number categories, serial number category 1, serial number category 2 and serial number category 3, serial number category 1 represents the highest level question number, serial number category 2 represents the next level question number of serial number category 1, and serial number category 3 represents the next level question number of serial number category 2. The question area corresponding to serial number category 1 and the subsequent question areas corresponding to serial number category 2 / serial number category 3 usually indicate that they belong to a large question, while the question area corresponding to serial number category 2 / serial number category 3 indicates the small questions under the large question.

[0077] After determining the final question area range corresponding to each question number, the question area corresponding to the highest level question number is subjected to text recognition and semantic classification to determine whether to frame a large question or a small question, thereby avoiding the situation where meaningless questions are framed. This embodiment can ensure that the results of the frame selection are complete, and plays a good supporting role in subsequent business. It is particularly suitable for the situation where test papers, exercise books, books, etc. are tilted or arched in the target image, and it solves the technical problems commonly existing in the existing question frame selection methods.

[0078] The above-mentioned embodiment is improved to obtain the present embodiment. In the step S20 of the present embodiment, the detection of the question numbers at each level in the target image, the question areas corresponding to the question numbers and the question row areas further includes: S21 locating and identifying the question numbers at each level in the target image to obtain the question numbers at each level; S22 treating the question areas corresponding to the question numbers at each level as a whole and dividing them along the edge of the question to obtain the question areas corresponding to the question numbers; S23 dividing each line of text in the question as a whole to obtain the question row area.

[0079] In this embodiment, after obtaining the target image, different models are created to detect the question numbers at each level in the target image, the question areas corresponding to each question number, and the question row areas. It should be noted that there is no order of precedence for the detection of the question numbers, the question areas corresponding to each question number, and the question row areas, and there is no dependency between them. Different AI models are used to process the target images independently.

[0080] The detection methods of the title area, title number and title line should be applicable to the situation where the title distribution in the target image is relatively complex, such as tilt and arch. For example, in the example, AI models such as MS-RCNN and PA-NET are used to detect the title area in the target image. AI models such as faster-RCNN and yolo (You Only LookOnce: Unified, Real-Time Object Detection) are used to locate and identify the title number in the target image. AI models such as TextSnake and PAN are used to detect the title line in the target image.

[0081] After creating different models to detect the question numbers at all levels in the target image, the question areas corresponding to the question numbers, and the question row areas, the question areas corresponding to the question numbers at all levels are finally determined by fusion. After determining the final question area range corresponding to each question number, the question area corresponding to the highest level question number is subjected to text recognition and semantic classification to determine whether it is a large question or a small question, so as to avoid the situation where meaningless questions are selected. The method provided in this embodiment can ensure that the results of the selection are complete, and it plays a good supporting role in subsequent business. It is particularly suitable for situations where test papers, exercise books, books, etc. are tilted or arched in the target image, and it solves the technical problems commonly existing in the existing question selection methods.

[0082] The present embodiment is obtained by improving the above-mentioned embodiment. In the present embodiment, step S30 of the detected question serial number, the question area corresponding to each question serial number and the question row area information are integrated to determine the final question area range corresponding to each question serial number, which further includes: S31 matching the question serial number and the question row area information with the question area information; S32 when mismatch information occurs, adjusting the question area according to the question serial number and / or the question row area information to determine the final question area range corresponding to each question serial number.

[0083] Since the features of the title number, the title area corresponding to each title number, and the title line information are extracted separately, there may be differences between them, and the purpose of the frame is to frame the title area corresponding to the title number. Therefore, in this embodiment, by creating different models, the title numbers of each level in the target image, the title area corresponding to each title number, and the title line area are detected respectively, and then the title area is taken as the core, and the title number and title line information are used to supplement it, so as to finally confirm the title area and improve the detection accuracy of the title area. In this way, when the detected title area is incomplete, the information in other titles is mistakenly framed, etc., it can be well adjusted according to the title number and title line area information, and the detected title area information is improved.

[0084] There is no specific limitation on the fusion algorithm here. For example, it can be implemented using AI models such as fasttext and bert. The adjustment rules can also be limited according to the actual situation. For example, when the detected serial numbers of the questions at all levels have a matching error with the detected question area information, it is further supplemented by the question row area information; for another example, when the detected question row area information has a matching error with the detected question area information, it is further supplemented by the serial number information of the questions at all levels; for another example, when the detected question area information does not match the detected question row area information and the serial number information of the questions at all levels, the detected question area information shall be used as the basis for confirmation, etc. Figure 2-4 It can be seen from the example that compared with the title area detection results and the title number detection results, the title row detection results have a wider range. Therefore, in the process of adjusting the title area, the detection results of the title number and the title row cannot be adjusted to the box selection results of the title area, and comprehensive consideration is required.

[0085] The present embodiment is obtained by improving the above-mentioned embodiment. In step S40 of the present embodiment, text recognition and semantic classification are performed on the question area corresponding to the highest-level question number to obtain the selection range mark of each question, including: S41 sending the question area corresponding to the highest-level question number to a text recognition tool for text recognition; S42 semantically classifying the recognized text, and calculating the correlation between the semantics based on the classification result; S43 comparing the calculated semantic correlation with a preset correlation threshold; S44 outputting the selection range mark of the question based on the comparison result.

[0086] In this embodiment, the question areas containing the question numbers of each level outputted in step S30 are individually framed, that is, the question areas corresponding to each level of question numbers are individually framed, but in actual applications, usually only the question areas that need to be answered are framed, so it is necessary to finally confirm whether to frame the question area corresponding to the highest level of question numbers. For example, for multiple-choice questions, the question area corresponding to the highest level of question numbers usually only includes the three words "multiple-choice questions", not the question content that needs to be answered, and it is not necessary to frame it in the process of framing the questions. Therefore, it is necessary to make a judgment on whether to frame it together.

[0087] Specifically, the question area of ​​the highest level question number is first sent to a text recognition tool such as OCR for text recognition; then the recognized text information is sent to the AI ​​model for semantic classification, and the correlation between the text in the question area corresponding to the question number of this level is calculated, and then the selection range mark is output according to the correlation. The semantic analysis algorithm is also not specifically limited here, such as using AI models such as fasttext to achieve the purpose.

[0088] In this embodiment, the frame selection range mark includes: a mark for frame selection including the question area corresponding to the highest level question number and a mark for frame selection not including the question area corresponding to the highest level question number. Specifically, when the calculated semantic relevance is less than the preset relevance threshold, it is determined that the semantic relevance between the characters in the question area corresponding to the question number of this level is not high, that is, the content in the question area is not actually the question content that needs to be answered, and has no substantial meaning for solving the problem, then the mark for frame selection not including the question area corresponding to the highest level question number is output, and then the question areas of the question numbers of other levels except the highest level question number in the question are framed according to the mark (the question areas corresponding to the question numbers of other levels are framed as a whole). When the calculated semantic relevance is greater than or equal to the preset relevance threshold, it is determined that the semantic relevance between the characters in the question area corresponding to the question number of this level is high, that is, the content in the question area is the question content that needs to be answered, then the mark for frame selection including the question area corresponding to the highest level question number is output, and then the question area including the question area corresponding to the question number of the highest level is framed according to the mark (the question areas corresponding to the question numbers of all levels in the question are framed as a whole). The correlation threshold can be determined according to the actual application, such as being set to 0.5, 0.6 or even larger. After obtaining the frame selection range mark, the question areas of each level of question sequence number are deleted or merged to present the final result.

[0089] In one example, assume that the question to be selected is a fill-in-the-blank question, corresponding to the first-level title number 1. The fill-in-the-blank question contains 4 sub-questions, corresponding to the second-level title numbers 1.1, 1.2, 1.3 and 1.4 respectively. After text recognition and semantic classification, the calculated semantic correlation is less than the preset correlation threshold 0.5, then the output is a mark that does not contain the question area corresponding to the highest-level question number. Subsequently, according to the mark, the question areas corresponding to the 4 second-level title numbers 1.1, 1.2, 1.3 and 1.4 in the question are selected as a whole.

[0090] In this embodiment, by performing text recognition and semantic classification on the question area corresponding to the highest level question number, a corresponding selection range mark is output, and the question area is subsequently selected according to the selection range mark, thereby avoiding the occurrence of incorrect selection to a great extent. In particular, when processing a question containing multiple levels of question numbers, the required question area can be correctly selected.

[0091] In one example, the detection result of the title area of ​​the target image is as follows: Figure 2As shown in the figure, it can be seen that the target image includes 4 application questions. The first-level question number of question 1 (indicated by "1") includes 3 second-level question numbers (indicated by "(1)", "(2)" and "(3)" respectively). The first-level question numbers of questions 2 and 3 (indicated by "2" and "3" respectively) do not include second-level question numbers. The first-level question number of question 4 (indicated by "4") includes 2 second-level question numbers (indicated by "(1)" and "(2)" respectively). This process selects the question areas of each level of question number separately, that is, the question areas of the stem part and each question part are selected separately. For example, in question 1, the question areas corresponding to the first-level question number and the three second-level question numbers are selected separately. The detection results of the question numbers are shown in the figure. Figure 3 As shown, the identified question number 12 includes the first-level question numbers 1, 2, 3 and 4, and the second-level question numbers (1), (2) and (3). Figure 4 As shown in FIG. 1 , a corresponding question row area 13 is obtained by selecting each line of text in the question. The picture sent to the text recognition tool is as follows: Figure 5 As shown, only the question area 14 corresponding to the highest level question number (the first level question number, only two question areas are shown in the figure as an example) is subjected to text recognition and semantic classification. The result of selecting each question in the picture according to the selection range mark is shown in FIG. Figure 6 As shown, the selected questions are all large questions, and the question area containing the higher-level question number and the question area of ​​the lower-level question number are selected as a whole.

[0092] like Figure 7 As shown, the present invention also provides a framing device 100 for student homework scenarios, including: an image acquisition module 110, used to acquire a target image containing a topic to be framed; a detection module 120, used to detect the topic numbers of each level in the target image acquired by the image acquisition module, the topic area corresponding to each topic number, and the topic row area; an information fusion module 130, used to fuse the topic numbers, the topic areas corresponding to each topic number, and the topic row area information detected by the detection module, to determine the final topic area range corresponding to each topic number; a semantic analysis module 140, used to perform text recognition and semantic classification on the topic area corresponding to the highest level topic number determined by the information fusion module, to obtain a framing range mark for each topic; a topic framing module 150, used to frame each topic in the image according to the framing range mark output by the semantic analysis module.

[0093] In this embodiment, after obtaining the target image, the detection module 120 detects the question numbers at each level, the question areas corresponding to each question number, and the question row areas in the target image by creating different models. It should be noted that there is no order of precedence for the detection of the question numbers, the question areas corresponding to each question number, and the question row areas, and there is no dependency between them. Different AI models are used to process the target images independently.

[0094] The detection methods of the title area, title number and title line should be applicable to the situation where the title distribution in the target image is relatively complex, such as tilt and arch. For example, in the example, AI models such as MS-RCNN and PA-NET are used to detect the title area in the target image. AI models such as faster-RCNN and yolo (You Only LookOnce: Unified, Real-Time Object Detection) are used to locate and identify the title number in the target image. AI models such as TextSnake and PAN are used to detect the title line in the target image.

[0095] In one example, the detection result of the title area of ​​the target image is as follows: Figure 2 As shown in the figure, it can be seen that the target image includes 4 application questions. The first-level question number of question 1 (indicated by "1") includes 3 second-level question numbers (indicated by "(1)", "(2)" and "(3)" respectively). The first-level question numbers of questions 2 and 3 (indicated by "2" and "3" respectively) do not include second-level question numbers. The first-level question number of question 4 (indicated by "4") includes 2 second-level question numbers (indicated by "(1)" and "(2)" respectively). This process selects the question areas of each level of question number separately, that is, the question areas of the stem part and each question part are selected separately. For example, in question 1, the question areas corresponding to the first-level question number and the three second-level question numbers are selected separately. The detection results of the question numbers are shown in the figure. Figure 3 As shown, the identified question number 12 includes the first-level question numbers 1, 2, 3 and 4, and the second-level question numbers (1), (2) and (3). Figure 4 As shown in the figure, it can be seen that for each line of text in the title, a corresponding title line area 13 is obtained by selecting a frame.

[0096] After the detection module 120 detects the question numbers at all levels, the question areas corresponding to the question numbers, and the question row areas in the target image by creating different models, the information fusion module 130 then finally determines the question areas corresponding to the question numbers at all levels by means of fusion. After determining the final question area range corresponding to each question number, the question area corresponding to the highest level question number is subjected to text recognition and semantic classification to determine whether it is a large question or a small question, thereby avoiding the situation where meaningless questions are selected. The method provided in this embodiment can ensure that the results of the selection are complete, and plays a good supporting role in subsequent business. It is particularly suitable for situations where test papers, exercise books, books, etc. are tilted or arched in the target image, and it solves the technical problems commonly found in existing question selection methods.

[0097] The above-mentioned embodiment is improved to obtain the present embodiment, in which the detection module 120 includes: a question number detection unit, used to locate and identify the question numbers at each level in the target image, and obtain the question numbers at each level; a question area detection unit, used to treat the question areas corresponding to the question numbers at each level as a whole and split them along the edge of the question, and obtain the question areas corresponding to each question number; a question line detection unit, used to split each line of text in the question as a whole, and obtain the question line area.

[0098] In this embodiment, after obtaining the target image, different models are created to detect the question numbers at each level in the target image, the question areas corresponding to each question number, and the question row areas. It should be noted that there is no order of precedence for the detection of the question numbers, the question areas corresponding to each question number, and the question row areas, and there is no dependency between them. Different AI models are used to process the target images independently.

[0099] The detection methods of the title area, title number and title line should be applicable to the situation where the title distribution in the target image is relatively complex, such as tilt and arch. For example, in the example, AI models such as MS-RCNN and PA-NET are used to detect the title area in the target image. AI models such as faster-RCNN and yolo (You Only LookOnce: Unified, Real-Time Object Detection) are used to locate and identify the title number in the target image. AI models such as TextSnake and PAN are used to detect the title line in the target image.

[0100] After creating different models to detect the question numbers at all levels in the target image, the question areas corresponding to the question numbers, and the question row areas, the question areas corresponding to the question numbers at all levels are finally determined by fusion. After determining the final question area range corresponding to each question number, the question area corresponding to the highest level question number is subjected to text recognition and semantic classification to determine whether it is a large question or a small question, so as to avoid the situation where meaningless questions are selected. The method provided in this embodiment can ensure that the results of the selection are complete, and it plays a good supporting role in subsequent business. It is particularly suitable for situations where test papers, exercise books, books, etc. are tilted or arched in the target image, and it solves the technical problems commonly existing in the existing question selection methods.

[0101] The above-mentioned embodiment is improved to obtain the present embodiment, in which the information fusion module 130 includes: an information matching unit, used to match the question serial number and question row area information with the question area information; and an area adjustment unit, used to adjust the question area according to the question serial number and / or question row area information when mismatched information occurs, and determine the final question area range corresponding to each question serial number.

[0102] Since the features of the title number, the title area corresponding to each title number, and the title line information are extracted separately, there may be differences between them, and the purpose of the frame is to frame the title area corresponding to the title number. Therefore, in this embodiment, by creating different models, the title numbers of each level in the target image, the title area corresponding to each title number, and the title line area are detected respectively, and then the title area is taken as the core, and the title number and title line information are used to supplement it, so as to finally confirm the title area and improve the detection accuracy of the title area. In this way, when the detected title area is incomplete, the information in other titles is mistakenly framed, etc., it can be well adjusted according to the title number and title line area information, and the detected title area information is improved.

[0103] There is no specific limitation on the fusion algorithm here. For example, it can be implemented using AI models such as fasttext and bert. The adjustment rules can also be limited according to the actual situation. For example, when the detected serial numbers of the questions at all levels have a matching error with the detected question area information, it is further supplemented by the question row area information; for another example, when the detected question row area information has a matching error with the detected question area information, it is further supplemented by the serial number information of the questions at all levels; for another example, when the detected question area information does not match the detected question row area information and the serial number information of the questions at all levels, the detected question area information shall be used as the basis for confirmation, etc. Figure 2-4It can be seen from the example that compared with the title area detection results and the title number detection results, the title row detection results have a wider range. Therefore, in the process of adjusting the title area, the detection results of the title number and the title row cannot be adjusted to the box selection results of the title area, and comprehensive consideration is required.

[0104] The present embodiment is obtained by improving the above-mentioned embodiment, wherein the semantic analysis module 140 includes a text recognition unit, which is used to send the question area corresponding to the highest-level question number into the text recognition tool for text recognition; a semantic classification unit, which is used to semantically classify the text recognized by the text recognition unit, and calculate the association between the semantics according to the classification result; a comparison unit, which is used to compare the semantic association calculated by the semantic classification unit with a preset association threshold; and a result output unit, which is used to output the box selection range mark of the question according to the comparison result of the comparison unit.

[0105] In this embodiment, the question areas containing the question numbers of each level outputted in step S30 are individually framed, that is, the question areas corresponding to each level of question numbers are individually framed, but in actual applications, usually only the question areas that need to be answered are framed, so it is necessary to finally confirm whether to frame the question area corresponding to the highest level of question numbers. For example, for multiple-choice questions, the question area corresponding to the highest level of question numbers usually only includes the three words "multiple-choice questions", not the question content that needs to be answered, and it is not necessary to frame it in the process of framing the questions. Therefore, it is necessary to make a judgment on whether to frame it together.

[0106] Specifically, the question area of ​​the highest level question number is first sent to a text recognition tool such as OCR for text recognition; then the recognized text information is sent to the AI ​​model for semantic classification, and the correlation between the text in the question area corresponding to the question number of this level is calculated, and then the selection range mark is output according to the correlation. The semantic analysis algorithm is also not specifically limited here, such as using AI models such as fasttext to achieve the purpose.

[0107] In this embodiment, the frame selection range mark includes: a mark that the frame selection includes the question area corresponding to the highest level question number and a mark that the frame selection does not include the question area corresponding to the highest level question number. Specifically, when the calculated semantic relevance is less than the preset relevance threshold, it is determined that the semantic relevance between the characters in the question area corresponding to the question number of this level is not high, that is, the content in the question area is not actually the question content that needs to be answered, and has no substantive meaning for solving the problem, then the output frame selection does not include the mark of the question area corresponding to the highest level question number. When the calculated semantic relevance is greater than or equal to the preset relevance threshold, it is determined that the semantic relevance between the characters in the question area corresponding to the question number of this level is high, that is, the content in the question area is the question content that needs to be answered, then the output frame selection includes the mark of the question area corresponding to the highest level question number. The relevance threshold can be determined according to the actual application, such as being set to 0.5, 0.6 or even larger. After obtaining the frame selection range mark, the question areas of the question numbers at each level are deleted or merged to present the final result.

[0108] In this embodiment, by performing text recognition and semantic classification on the question area corresponding to the highest level question number, a corresponding selection range mark is output, and the question area is subsequently selected according to the selection range mark, thereby avoiding the occurrence of incorrect selection to a great extent. In particular, when processing a question containing multiple levels of question numbers, the required question area can be correctly selected.

[0109] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned program modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program units or modules to complete all or part of the functions described above. The program modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a processing unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software program unit. In addition, the specific names of the program modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0110] Figure 82 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention. As shown, the terminal device 200 includes: a processor 220, a memory 210, and a computer program 211 stored in the memory 210 and executable on the processor 220, for example: a frame question association program in a student homework scenario. When the processor 220 executes the computer program 211, the steps in the frame question method embodiments in each student homework scenario are implemented, or when the processor 220 executes the computer program 211, the functions of each module in the frame question device embodiment in the student homework scenario are implemented.

[0111] The terminal device 200 may be a tutoring machine, a learning machine, a mobile phone or a tablet computer with a learning function, or other smart device. The terminal device 200 may include, but is not limited to, a processor 220 and a memory 210. Those skilled in the art will appreciate that Figure 8 It is only an example of the terminal device 200 and does not constitute a limitation of the terminal device 200. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 200 may also include input and output devices, display devices, network access devices, buses, etc.

[0112] The processor 220 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor 220 may be a microprocessor or any conventional processor, etc.

[0113] The memory 210 may be an internal storage unit of the terminal device 200, such as a hard disk or memory of the terminal device 200. The memory 210 may also be an external storage device of the terminal device 200, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 200. Further, the memory 210 may also include both an internal storage unit of the terminal device 200 and an external storage device. The memory 210 is used to store a computer program 211 and other programs and data required by the terminal device 200. The memory 210 may also be used to temporarily store data that has been output or is to be output.

[0114] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0115] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed with hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0116] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0117] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0118] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0119] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by sending instructions to related hardware through the computer program 211. The computer program 211 can be stored in a computer-readable storage medium. When the computer program 211 is executed by the processor 220, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program 211 includes: computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device that can carry the computer program 211 code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in computer-readable storage media can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable media do not include electrical carrier signals and telecommunications signals.

[0120] It should be noted that the above embodiments can be freely combined as needed. The above are only preferred embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A method for framing questions in a student homework scenario, characterized in that: include: Get the target image containing the title to be selected; Detecting the question numbers at each level in the target image, the question areas corresponding to the question numbers, and the question row areas respectively; The detected question numbers, the question areas corresponding to the question numbers, and the question row area information are integrated to determine the final question area range corresponding to the question numbers; Perform text recognition and semantic classification on the question area corresponding to the highest level question number to obtain the selection range mark of each question; Send the question area corresponding to the highest level question number to a text recognition tool for text recognition; semantically classify the recognized text, and calculate the correlation between the semantics based on the classification results; comparing the calculated semantic relevance with a preset relevance threshold; Outputting a selection range mark of the question according to the comparison result; the selection range mark includes: a mark for selecting a question area corresponding to the highest level question number and a mark for selecting a question area not corresponding to the highest level question number; When the calculated semantic relevance is less than the preset relevance threshold, that is, the content in the question area is not actually the question content that needs to be answered, the output box selection does not include the mark of the question area corresponding to the highest level question number; when the calculated semantic relevance is greater than or equal to the preset relevance threshold, that is, the content in the question area is the question content that needs to be answered, the output box selection includes the mark of the question area corresponding to the highest level question number; Each title in the picture is framed and selected according to the frame selection range mark.

2. The framing method according to claim 1, characterized in that: The detecting of the question numbers at each level in the target image, the question areas corresponding to the question numbers, and the question row areas respectively includes executing the following steps in parallel: Locate and identify the serial numbers of each level of questions in the target image to obtain the serial numbers of each level of questions; The question areas corresponding to the question numbers at each level are treated as a whole and divided along the edge of the question to obtain the question areas corresponding to the question numbers; Split each line of text in the title as a whole to obtain the title line area.

3. The framing method according to claim 1 or 2, characterized in that: The detected question numbers, the question areas corresponding to the question numbers, and the question row area information are integrated to determine the final question area range corresponding to the question numbers, including: Matching the topic serial number and topic row area information with the topic area information; When mismatch information appears, the topic area is adjusted according to the topic serial number and / or topic row area information to determine the final topic area range corresponding to each topic serial number.

4. A frame question device in a student homework scenario, characterized in that: include: An image acquisition module is used to acquire a target image containing the title to be framed; A detection module, used to detect the question numbers of each level, the question areas corresponding to the question numbers, and the question row areas in the target image acquired by the image acquisition module; An information fusion module is used to fuse the question numbers, question areas corresponding to each question number, and question row area information detected by the detection module to determine the final question area range corresponding to each question number; A semantic analysis module, used to perform text recognition and semantic classification on the question area corresponding to the highest level question number determined by the information fusion module, and obtain a selection range mark of each question; A title selection module, used to select each title in the picture according to the selection range mark output by the semantic analysis module; The semantic analysis module includes: A character recognition unit, used for sending the question area corresponding to the highest level question number to a character recognition tool for character recognition; A semantic classification unit, used to semantically classify the characters recognized by the character recognition unit, and calculate the correlation between the semantics according to the classification results; A comparison unit, used for comparing the semantic relevance calculated by the semantic classification unit with a preset relevance threshold; A result output unit, used for outputting a selection range mark of the question according to the comparison result of the comparison unit; The frame selection range mark includes: a mark for frame selection including the question area corresponding to the highest level question number and a mark for frame selection not including the question area corresponding to the highest level question number; When the comparison unit determines that the semantic relevance is less than the preset relevance threshold, that is, the content in the question area is not actually the content of the question to be answered, the result output unit outputs a mark indicating that the box selection does not include the question area corresponding to the highest level question number; When the comparison unit determines that the semantic relevance is greater than or equal to the preset relevance threshold, that is, the content in the question area is the content of the question that needs to be answered, the result output unit outputs a mark of the question area corresponding to the highest level question number.

5. The frame title device according to claim 4, characterized in that: The detection module includes: The question number detection unit is used to locate and identify the question numbers of each level in the target image to obtain the question numbers of each level; A question area detection unit is used to divide the question areas corresponding to the question numbers at each level as a whole and along the question edges to obtain the question areas corresponding to the question numbers; The title line detection unit is used to segment each line of text in the title as a whole to obtain the title line area.

6. The frame title device according to claim 4 or 5, characterized in that: The information fusion module includes: An information matching unit, used for matching the topic sequence number and topic row area information with the topic area information; The area adjustment unit is used to adjust the topic area according to the topic serial number and / or topic row area information when mismatch information appears, and determine the final topic area range corresponding to each topic serial number.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor runs the computer program, the steps of the framing method in the student homework scenario as described in any one of claims 1-3 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the framing method in the student homework scenario as described in any one of claims 1 to 3 are implemented.