Examination paper question splitting method and device based on abstract syntax tree

Abstract

The invention discloses a test paper question splitting method and device based on an abstract syntax tree, and the method comprises the following steps: S1, reading and analyzing an original material, and converting the original material into an HTML (Hypertext Markup Language) fragment; s2, analyzing p labels generated in the conversion one by one, extracting plain text contents, removing front and back spaces, and converting the plain text contents into Token required by the method; s3, converting the Tokens generated in the previous step in an abstract syntax tree one by one; and S4, traversing the syntax tree by adopting preorder traversal to generate structured topic information. Compared with the prior art, the system has the advantages of high efficiency, accuracy, flexibility, clear structure and easiness in maintenance.

Description

Technical Field

[0001] The present invention relates to the field of educational digital technology, and in particular to a test paper question splitting method and device based on an abstract syntax tree. Background Art

[0002] With the in-depth development of educational informatization, quickly and accurately converting test content in the form of paper test papers, Word documents, PDF files, etc. into structured data has become an important part of the process of educational digitization. Traditional methods such as manual input and OCR recognition after scanning have problems such as large workload, low efficiency, and easy errors, which are difficult to meet the needs of large-scale and efficient test paper processing. Therefore, it is particularly important to develop an automated and intelligent test paper question splitting method. Summary of the invention

[0003] In view of the low efficiency and error-proneness of digital processing of test paper content in the prior art, the present invention proposes a test paper question splitting method and device based on an abstract syntax tree (AST). The method automatically parses the test paper content and converts it into a structured data format, effectively improving the efficiency and accuracy of test paper processing.

[0004] Specifically, the technical solution provided by the present invention is: a test paper question splitting method and device based on an abstract syntax tree, comprising the following steps:

[0005] S1, read, parse the original material, and convert it into HTML fragments;

[0006] S1.1. Convert Word documents into p tags in paragraph units;

[0007] S1.2, bold is converted to b tag, italic is converted to i tag, underline is converted to u tag, table is converted to table tag; font and font size elements are stored in style attribute of span tag;

[0008] S1.3, "Number" shall be converted into corresponding text serial number according to Word formatting rules;

[0009] S1.4. Convert the embedded image to a picture file and save it as a picture file, and reference it with the img tag in the HMTL fragment;

[0010] S1.5, embed OLE objects, extract the images of OLE objects and save them as image files, and reference them in HTML fragments using img tags;

[0011] S1.6. The embedded formula is converted to LateX format and used as the data-latex custom attribute value of the span tag;

[0012] S1.7. First, convert the PDF document into a Word document and then convert it according to the Word document rules;

[0013] S1.8. Use OCR technology to convert paper materials into text versions, organize the content by line, and convert the lines into p tags;

[0014] S2. Parse each p tag generated in the above conversion one by one, extract the pure text content, remove the leading and trailing spaces, and then convert it into the Tokens required by the method;

[0015] S2.1. For the first n lines whose content contains the key information of "Final Test", "Mid-Term Test", "Quality Inspection", and "Monthly Exam", mark them as TitleToken;

[0016] S2.2. If it can match the regular expression "^[一二三四五六七八九十]+、", mark it as SectionToken;

[0017] S2.3. If it can match the regular expression "^(?P <no>\d+)[{DOT_SPACE}]+.*”, marked as QuestionToken;

[0018] S2.4. It can match the regular expression "([A-Z][..])+” and is marked as OptionToken;

[0019] S2.5. It can match the regular expression "([

]?Answer[:|\s|

[0020] S2.6. It can match the regular expression "^([

]?Analysis|Explanation[:|\s|

[0021] S2.7. It can match the regular expression

[0022] "^(?P <start>[{LEFT_PARENTHESIS}]?)(?P <no>\d+)(?P <end>[{RIGHT_PARENTHESIS}]+)(?P<conten t> .*)", marked as SubQuestion Token;

[0023] S2.8, the first n lines containing "Paper Marking", "Score", "Top Secret★Before Activation", "Exam Number:", "Exam Scope", "Exam Time", "Notes" are marked as, Ignore Token;

[0024] S2.9, if none of the above matches, it is marked as a normal text TextToken;

[0025] S3, transform the tokens generated in the above steps into the abstract syntax tree one by one;

[0026] S3.1. When the SectionToken node is read, the section conversion algorithm is executed:

[0027] S3.1.1, read the next node until the next node is not a TextToken;

[0028] S3.1.2. Store all the read Text nodes in sequence into secitonToken.children;

[0029] S3.1.3, read the next node until the next node is not QuestionToken;

[0030] S3.1.4, store the read QuestionToken nodes in sectionToken.cildren in order;

[0031] S3.1.5. Insert the Text node in the SectionToken child node Children into position 0 of the first questionToken.children. This step is called text push-down, which can transfer additional materials into the question stem.

[0032] S3.2. After reading the QuestionToken tag, execute the question conversion algorithm:

[0033] S3.2.1, read the next node until the next node is not a TextToken;

[0034] S3.2.2. Store all the read Text nodes into question.chidlren in order;

[0035] S3.2.3. Read the next node until the next node is not OptionToken;

[0036] S3.2.4. Separate multiple options from all OptionToken nodes and store them in question.children in order.

[0037] S3.2.5, the next node is AnswerToken or AnalysisToken, stored in quesiton.children;

[0038] S3.2.6, if the next node is SubQuestionToken, check the sub-question mark symbol in the current quesiton.children. If they are the same, parse the sub-question according to the algorithm similar to QuestionToken and store it in quesion.children; if the sub-question symbols are different, convert them into TextToken and store them in the current token.children;

[0039] S3.3, read the IgnoreToken tag, and then skip the Token;

[0040] S4, traversing the above syntax tree using pre-order traversal to generate structured question information;

[0041] S4.1, SectionToken node, directly traverse its Children node;

[0042] S4.2, QuestionToken node processing method:

[0043] S4.2.1. Filter all TextTokens in children and organize them into question stems;

[0044] S4.2.2, filter all OptionTokens in children and organize them into options;

[0045] S4.2.3. Filter all SubQuestionTokens in children as the sub-questions of the current question; process the stem, options, and answers of the sub-questions according to the QuestionToken processing method;

[0046] S4.2.4, filter the AnswerToken in children as the reference answer to the question;

[0047] S4.2.5, filter AnalysisToken in children as the question analysis;

[0048] S4.2.6. Store the generated structured data into the result list.

[0049] The advantages of the present invention compared with the prior art are: 1) high efficiency: through automated parsing and conversion, manual intervention is greatly reduced, and the efficiency of test paper content processing is improved;

[0050] 2) Accuracy: The use of precise matching technologies such as regular expressions ensures accurate classification and marking of test paper content, reducing the error rate

[0051] 3) Flexibility: Supports input of original materials in multiple formats (such as Word documents, PDF documents, paper materials, etc.), with strong adaptability and flexibility;

[0052] 4) Clear structure: By constructing an abstract syntax tree, a hierarchical and structured representation of the test paper content is achieved, which is convenient for subsequent processing and analysis;

[0053] 5) Easy to maintain: The method and device of the present invention have clear structure, simple logic, and are easy to maintain and expand. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flow chart of the patent of this invention. DETAILED DESCRIPTION

[0055] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0056] When the present invention is specifically implemented, Figure 1 As shown, this program mainly includes:

[0057] 1. Read and parse the original material: First, read and parse the original material (such as Word documents, PDF documents, paper materials, etc.) and convert it into HTML fragments. During the conversion process, according to the different elements in the material (such as text format, pictures, tables, formulas, etc.), use corresponding HTML tags to mark them to ensure the integrity and accuracy of the information.

[0058] 2. Tokenization: Parse the converted HTML fragments one by one, extract the plain text content and remove the leading and trailing spaces, and then convert the text content into different types of tokens (such as TitleToken, Section Token, Question Token, Option Token, etc.) according to the preset regular expression rules. This step realizes the preliminary classification and marking of the test paper content.

[0059] 3. Construct an abstract syntax tree (AST): Convert the generated tokens into nodes in the abstract syntax tree one by one. According to the type of node (such as SectionToken, QuestionToken, etc.), execute the corresponding conversion algorithm to construct a syntax tree with a clear structure and distinct levels. During the construction process, through strategies such as text push-down, the additional materials are reasonably transferred to the question stem to ensure the integrity of the question information.

[0060] 4. Generate structured question information: Use pre-order traversal to traverse the above syntax tree and generate structured question information based on the node type and attributes. For each QuestionToken node, extract its stem, options, sub-questions, answers, and analysis information, organize this information into a structured data format, and store it in the result list.

[0061] Embodiment 2

[0062] Based on the first embodiment, Figure 1 As shown, this embodiment provides a specific description of the test paper parsing method:

[0063] First, convert the original material into HTML fragments; the test papers used for teaching contain a variety of information sources, including Word documents, PDF documents, and paper materials. In the process of converting the original material into HTML, different conversion methods need to be used according to different material formats.

[0064] Word documents can be directly read using existing third-party libraries. Then, based on the document content and format, they are converted according to the following rules:

[0065] (1) The paragraph is converted to a p tag, the paragraph text is contained in a span tag, and the style is processed according to the downlink rules;

[0066] (2) Bold is converted to the b tag, italics is converted to the i tag, underline is converted to the u tag, and tables are converted to the table tag; font, font size, etc. are stored in the style attribute of the span tag;

[0067] (3) "Number" is converted into the corresponding text serial number according to Word format rules;

[0068] (4) The embedded image is converted and saved as an image file and referenced using the img tag in the HTML fragment;

[0069] (5) Embed OLE objects, extract images of OLE objects and save them as image files, and reference them in HTML fragments using img tags;

[0070] (6) The embedded formula is converted to LateX format and used as the data-latex custom attribute value of the span tag.

[0071] For PDF documents, automatically call the "Save As" function of Word. After converting the PDF into a Word document, parse it according to Word rules;

[0072] For paper documents, use OCR to count and convert them into text format, and process them according to the following rules / steps:

[0073] (1) Build an OCR Pipeline using layout detection, text detection, orientation recognition, character recognition, and table recognition;

[0074] (2) In layout detection, save the positions and images of the detected images;

[0075] (3) Before text detection, mask the images and page footer content detected in layout detection;

[0076] (4) Organize the results after text conversion into "lines" according to character coordinates; use sup and sub tags for superscripts and subscripts.

[0077] Example 3

[0078] Based on Example 2, as Figure 1 shown, identify and mark the paragraph types to form Tokens. In this example, the purpose of identifying and marking the paragraph types is to identify the type of each line without considering the context, laying a foundation for the next step of constructing an abstract syntax tree. Perform a single pass of line-by-line (p) scanning on the HTML fragments generated in the above steps, and classify the p tags according to the following rules:

[0079] (1) For the first n lines whose content contains key information such as "Final Quiz", "Mid-Term Quiz", "Quality Inspection", "Monthly Exam", etc., mark them as Title;

[0080] (2) Those that can match the regular expression "^[一二三四五六七八九十]+、" are marked as Section;

[0081] (3) Those that can match the regular expression "^(?P <no>\d+)[{DOT_SPACE}]+.*”, marked as the question Question;

[0082] (4) Can match the regular expression "([A-Z][..])+”, marked as the option Option;

[0083] (5) Can match the regular expression "([

]?Answer[:|\s|

[0084] (6) Can match the regular expression "^([

]?Analysis|Detailed Explanation[:|\s|

[0085] (7) Can match the regular expression

[0086] "^(?P <start>[{LEFT_PARENTHESIS}]?)(?P <no>\d+)(?P <end>[{RIGHT_PARENTHESIS}]+)(?P <content>.*)”, marked as a sub-question SubQuestion;

[0087] (8) The first n lines, which contain information such as "Paper Marking", "Score", "Top Secret★Before Activation", "Exam Number:", "Exam Scope", "Exam Time", "Notes", etc., are marked as Ignore;

[0088] (9) If none of the above matches, it will be marked as normal text.

[0089] Embodiment 4

[0090] Based on the third embodiment, Figure 1 As shown, the abstract syntax tree of the test paper is generated. In this embodiment, the test paper is abstracted into a tree structure. The stem, options, answers, and analysis are used as leaf nodes, the questions, sub-questions, and sections are used as intermediate nodes, and the title is used as the root node. According to the following rules, the recursive descent algorithm is used to traverse the token list generated in the above steps one by one:

[0091] (1) First, create two variables, token and nextToken, which represent the current node to be processed and the next node respectively. When initialized, token points to Null and nextToken points to the first node.

[0092] (2) "Step forward" operation, token and nextToken move forward one step respectively. At this time, token = nextToken; nextToken points to the next new node;

[0093] (3) When the next node is Title, use token as the root node of the syntax tree and "step forward";

[0094] (4) When the next node is Section:

[0095] - Further check the question scores and scoring rules in the Section and store them in the section attributes;

[0096] -Query the subject question type table, establish the preprocessor and postprocessor, and save them in the section attributes;

[0097] -Use section as a child node of the root node of the syntax tree;

[0098] - "Step forward";

[0099] (5) If the next node is a TextToken, store the TextToken in section.Children and repeat this step by "stepping forward" until the next node is not a TextToken;

[0100] (6) If the next node is a QuestionToken, extract the question and store it in section.Children until the next question is no longer a QuestionToken;

[0101] (7) After processing each question, the following will be executed: If section contains preprocessor attributes, the text push-down algorithm will be executed: extract all TextTokens in section.Children and insert them into the head of the current question.Children, remove the pushed-down TextTokens from section.Children; set question to a question group;

[0102] (8) After processing each question, the following will be executed: If the section contains a post-processor attribute, the post-processor program will be executed. The post-processor can be: split options; automatically create sub-questions. Split options correspond to English subjects, multiple choice fill-in-the-blank scenarios, and sub-questions are followed by options (such as: 7.A.About B.about C.How aboutD.What about); automatically create sub-questions to create a sub-question scenario for each blank;

[0103] (9) When the next node is a question, execute the question extraction algorithm:

[0104] (10) If the next node is a TextToken, store the token in question.Children and repeat this step by "stepping forward" until the next node is not a TextToken;

[0105] (11) If the next node is an OptionToken, store the token in question.Children and repeat this step by "stepping forward" until the next node is not an OptionToken;

[0106] (12) If the next node is AnswerToken, store the token in question.Children and "step forward";

[0107] (13) If the next node is AnalysisToken, store the token in question.Children and "step forward";

[0108] (14) If the next node is a SubQuestionToken, extract the sub-questions using the same algorithm as the question parsing algorithm;

[0109] (15) If the current question is already a sub-question, convert the sub-question extracted in the previous step into a TextToken and store it in question.Children;

[0110] (16) Check the sub-question symbols extracted in the previous two steps. If they are the same as the sub-question symbols already in question, the parsing is considered correct. Otherwise, they are directly stored in question.Children; otherwise, they are converted into TextToken and stored in question.Children.

[0111] (17) Repeat the above three steps until the next node is no longer a SubQuestionToken;

[0112] (18) If the next node is IgnoreToken, execute "forward step" to skip.

[0113] Embodiment 5

[0114] Based on the fourth embodiment, Figure 1 As shown, the pre-order traversal of the syntax tree generates a structured question

[0115] In this embodiment, the created test paper syntax tree includes a tree structure with the title as the root node, sections, questions, and sub-questions as intermediate nodes, and question stems, options, answers, texts, etc. as leaf nodes. It is necessary to traverse the syntax tree according to the pre-order traversal algorithm to convert the questions into structured data for subsequent storage in the question bank.

[0116] (1) Traverse to the Section node and directly traverse its Children node;

[0117] -Traverse to the Question node processing method:

[0118] -Filter all TextTokens in Children and organize them into question stems;

[0119] -Filter all OptionTokens in Children and organize them into options;

[0120] -Filter all SubQuestionTokens in Children as sub-questions of the current question; process the stem, options, answers, etc. of the sub-questions according to the Question processing method;

[0121] (2) Filter AnswerToken in Children as the reference answer to the question;

[0122] (3) Filter AnalysisToken in Children as the question analysis;

[0123] (4) Store the generated structured data into the question list.

[0124] The following example of parsing the following text into a structured question is used to further illustrate this embodiment, which includes the following steps:

[0125] 1. Document parsing;

[0126] Document sources include Word, PDF, OCR, etc. In this example, it is assumed that the document has been parsed and the content is obtained:

[0127] People's Education Press (2019) High School Chemistry Second Semester Midterm Assessment Paper (Paper A)

[0128] 1. Single-choice questions

[0129] 1. Which of the following statements about chemical reaction rates is correct?

[0130] A. For any chemical reaction, the greater the reaction rate, the more obvious the reaction phenomenon

[0131] B. The rate of a chemical reaction usually expresses the decrease in the concentration of any reactant or the increase in the concentration of any product per unit time.

[0132]

[0133] C. The speed of a chemical reaction can be determined by the size of the chemical reaction rate.

[0134] Answer: C

[0135] Analysis: The greater the chemical reaction rate, the more obvious the reaction phenomenon is not necessarily. A is wrong. The concentration of solid or pure liquid is regarded as a constant. Generally, the change of the concentration of solid or pure liquid is not used to express the chemical reaction rate. B is wrong.

[0136] 2. Sulfuric acid is an important chemical raw material and can be used to produce fertilizers, pesticides, dyes, etc. For the following facts, the incorrect property of concentrated sulfuric acid in brackets is ()

[0137] A. Sucrose turns black after adding concentrated sulfuric acid (dehydration)

[0138] B. The solute mass fraction of concentrated sulfuric acid exposed to air becomes smaller (volatility)

[0139] C. Concentrated sulfuric acid reacts with charcoal under heating conditions (oxidizing property)

[0140] D. Concentrated sulfuric acid can be used to dry certain gases (water absorption)

[0141] Answer: B

[0142] Analysis: Concentrated sulfuric acid can remove H and O elements from organic matter such as sucrose, paper, cotton cloth and wood according to the composition ratio of water. This property belongs to the dehydration property of concentrated sulfuric acid. A is correct.

[0143] 2. Identify and mark paragraph types to form tokens;

[0144] Process each p one by one, and mark the corresponding row as the corresponding token according to the various token matching rules. The final tokens after marking are as follows:

[0145] TitleToken: People's Education Press (2019) High School Chemistry Second Semester Midterm Assessment Paper (Paper A)

[0146] SectionToken: 1. Single-choice questions

[0147] QuestionToken: 1. Which of the following statements about chemical reaction rates is correct?

[0148] OptionToken: A. For any chemical reaction, the greater the reaction rate, the more obvious the reaction phenomenon

[0149] OptionToken: B. The rate of a chemical reaction usually expresses the decrease in the concentration of any reactant or the increase in the concentration of any product per unit time.

[0150] OptionToken: C. The speed of a chemical reaction can be determined by the size of the chemical reaction rate.

[0151] AnswerToken: Answer: C

[0152] AnalysisToken: Analysis: The greater the chemical reaction rate, the more obvious the reaction phenomenon is not necessarily. A is wrong. The concentration of solid or pure liquid is regarded as a constant. Generally, the change of the concentration of solid or pure liquid is not used to express the chemical reaction rate. B is wrong.

[0153] QuestionToken: 2. Sulfuric acid is an important chemical raw material and can be used to produce fertilizers, pesticides, dyes, etc. For the following facts, the incorrect property of concentrated sulfuric acid in brackets is ()

[0154] OptionToken: A. Sucrose turns black after adding concentrated sulfuric acid (dehydration)

[0155] OptionToken: B. The solute mass fraction of concentrated sulfuric acid exposed to air becomes smaller (volatility)

[0156] OptionToken: C. Concentrated sulfuric acid reacts with charcoal under heating conditions (oxidizing property)

[0157] OptionToken: D. Concentrated sulfuric acid can be used to dry certain gases (water absorption)

[0158] AnswerToken: Answer: B

[0159] AnalysisToken: Analysis: Concentrated sulfuric acid can remove H and O elements from organic matter such as sucrose, paper, cotton cloth and wood according to the composition ratio of water. This property belongs to the dehydration property of concentrated sulfuric acid. A is correct.

[0160] 3. Generate the test paper syntax tree;

[0161] Each token is processed according to the rules for generating a syntax tree, and the final syntax tree generated is:

[0162] TitleToken

[0163] SectionToken

[0164] QuestionToken

[0165] OptionToken

[0166] OptionToken

[0167] OptionToken

[0168] AnswerToken

[0169] AnalysisToken

[0170] QuestionToken

[0171] OptionToken

[0172] OptionToken

[0173] OptionToken

[0174] OptionToken

[0175] AnswerToken

[0176] AnalysisToken

[0177] Embodiment 6

[0178] Based on Example 5, Figure 1 As shown, the syntax tree is traversed to generate structured questions. In this embodiment, after the syntax tree is generated, it is only necessary to follow the pre-order traversal algorithm to parse and transform the questions one by one in the same order as the test paper.

[0179] The above shows and describes the basic principles and main features of the present invention and the advantages of the invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.< / content> < / end> < / no> < / start> < / no> < / end> < / no> < / start> < / no>

Claims

1. A method and device for splitting test questions based on an abstract syntax tree, characterized in that it includes the following steps: S1. Read and parse the original materials and convert them into HTML fragments; S1.

1. Convert a Word document into p tags in units of paragraphs; S1.

2. Convert bold to b tags, italic to i tags, underline to u tags, and tables to table tags; Font and font size elements are stored in the style attribute of the span tag; S1.

3. Convert "numbering" into corresponding text serial numbers according to Word format rules; S1.

4. Embed images and keep them as image files, and reference them with img tags in the HMTL fragment; S1.

5. Embed OLE objects, extract the images of the OLE objects and save them as image files, and reference them with img tags in the HTML fragment; S1.

6. Convert embedded formulas into LaTeX format and use them as the data-latex custom attribute value of the span tag; S1.

7. First convert a PDF document into a Word document and convert it according to Word document rules; S1.

8. Use OCR technology to convert paper materials into text versions, organize the content in units of lines, and convert the lines into p tags; S2. Parse each p tag generated in the above conversion one by one, extract the pure text content, remove the leading and trailing spaces, and convert it into the Tokens required by the method; S2.

1. The content of the first n lines that contains key information such as "final test", "mid-term test", "quality inspection", and "monthly exam" is marked as Title Token; S2.

2. Those that can match the regular expression "^[one, two, three, four, five, six, seven, eight, nine, ten]+、” are marked as Section Token; S2.3, can match the regular expression "^(?P <no>\d+)[{DOT_SPACE}]+.*” is marked as QuestionToken; < / no> S2.

4. Those that can match the regular expression "([A-Z][..])+” are marked as Option Token; S2.

5. Those that can match the regular expression "([【]? Answer[:|\s|】])(.*)” are marked as AnswerToken; S2.

6. Those that can match the regular expression "^([【]? Analysis|Detailed Explanation[:|\s|】])(.*)” are marked as AnalysisToken; S2.

7. Those that can match the regular expression "^(？P <start>[{LEFT_PARENTHESIS}]?)(?P <no>\d+)(?P <end>[{RIGHT_PARENTHESIS}]+)(?P<content>.*)” are marked as SubQuestion Token; < / end> < / no> < / start> S2.

8. The first n lines that contain information such as "grading", "score", "top secret ★ before use", "exam number:", "exam scope", "exam time", and "matters needing attention" are marked as, Ignore Token; S2.

9. If none of the above match, it is marked as ordinary text TextToken; S3. Convert the Tokens generated in the above steps into an abstract syntax tree one by one; S3.

1. When reading the SectionToken node, execute the section conversion algorithm: S3.1.1, read the next node until the next node is not a TextToken; S3.1.

2. Store all the read Text nodes in sequence into secitonToken.children; S3.1.3, read the next node until the next node is not QuestionToken; S3.1.4, store the read QuestionToken nodes in sectionToken.cildren in order; S3.1.

5. Insert the Text node in the SectionToken child node Children into position 0 of the first questionToken.children. This step is called text push-down, which can transfer additional materials into the question stem. S3.

2. After reading the QuestionToken tag, execute the question conversion algorithm: S3.2.1, read the next node until the next node is not a TextToken; S3.2.

2. Store all the read Text nodes into question.chidlren in order; S3.2.

3. Read the next node until the next node is not OptionToken; S3.2.

4. Separate multiple options from all OptionToken nodes and store them in question.children in order. S3.2.5, the next node is AnswerToken or AnalysisToken, stored in quesiton.children; S3.2.6, if the next node is SubQuestionToken, check the sub-question mark symbol in the current quesiton.children. If they are the same, parse the sub-question according to the algorithm similar to QuestionToken and store it in quesion.children; if the sub-question symbols are different, convert them into TextToken and store them in the current token.children; S3.3, read the IgnoreToken tag, and then skip the Token; S4, traversing the above syntax tree using pre-order traversal to generate structured question information; S4.1, SectionToken node, directly traverse its Children node; S4.2, QuestionToken node processing method: S4.2.

1. Filter all TextTokens in children and organize them into the question stem; S4.2.2, filter all OptionTokens in children and organize them into options; S4.2.

3. Filter all SubQuestionTokens in children as the sub-questions of the current question; process the stem, options, and answers of the sub-questions according to the QuestionToken processing method; S4.2.4, filter the AnswerToken in children as the reference answer to the question; S4.2.5, filter AnalysisToken in children as the question analysis; S4.2.

6. Store the generated structured data into the result list.

Images