A method and system for segmenting science test paper text

Abstract

The present invention provides a method and system for segmenting science test paper text. The method obtains a database of historical test papers, randomly extracts various parts of the historical test papers in the database, and generates test paper text according to a preset typesetting format; identifies formulas and pictures in the test paper text, converts the formulas and pictures into corresponding characters, and performs sentence segmentation based on target symbols; automatically annotates each sentence, and inputs the annotated results into a constructed pointer annotation neural network model for training; inputs the test paper text to be segmented into a trained neural network model, and outputs a segmented test question list, wherein the sentences are encoded by modeling a longer contextual relationship, and a hierarchical attention mechanism from the word level to the sentence level is built to achieve deep semantic understanding, and the setting and interaction of the start position pointer and the end position pointer in the pointer annotation neural network model are used during model training to achieve more accurate test question text segmentation.

Description

Technical Field

[0001] The present invention belongs to the technical field of science test paper text segmentation, and in particular relates to a science test paper text segmentation method and system. Background Art

[0002] Text segmentation is a common task in NLP. In the field of education, the structuring of test questions is a necessary foundation for building an online question bank, and the structuring of test questions inevitably involves the segmentation of test paper text. Since test questions in actual scenarios are usually displayed and used in the form of test paper documents, if AI technology is to be used to analyze and personalize test questions in the test paper, each question in the test paper document needs to be cut out first. Due to the unique layout of test questions in the test paper, if one position is cut wrong during the cutting process, it will often affect the accuracy of cutting two questions. The length of test text varies, especially the test questions in the teacher's paper are often very long due to the existence of answers and analysis. In the case of limited resources, the current common text segmentation methods are still difficult to accurately segment long texts. Summary of the invention

[0003] Based on this, an embodiment of the present invention provides a method and system for segmenting science test paper text, aiming to solve the problem of inaccurate segmentation in the prior art when performing text segmentation on test questions of varying lengths and forms in science test papers.

[0004] A first aspect of an embodiment of the present invention provides a method for segmenting science test paper text, the method comprising:

[0005] Obtain a database of historical test papers, randomly extract parts of the historical test papers in the database, and generate test paper text according to a preset typesetting format, wherein in the process of generating the test paper text, preset marks are set correspondingly at the starting position and the ending position of each test question;

[0006] Identify formulas and pictures in the test paper text, convert the formulas and pictures into corresponding characters to obtain processed text, and then separate the test paper text into sentences based on target symbols;

[0007] According to the divided sentences in the processed text, each sentence is automatically annotated to obtain a sentence list and a corresponding label list of the test paper text, wherein the annotations of the sentences with the start position and the end position marks are different from those of other sentences;

[0008] Input the sentence list of the test paper text and the data in the corresponding label list into the constructed pointer annotation neural network model for training to obtain the target neural network model;

[0009] Input the test paper text to be segmented into the target neural network model, output the segmented test question list, and in the reasoning process of the target neural network model, use the adaptive sliding window strategy to perform output pointer sequence reasoning on the test paper text to be segmented, specifically including:

[0010] Obtaining a preset window length, and determining, according to the preset window length, a current text sequence selected by the preset window length, and an adjacent text sequence selected by the preset window length with the end of the current text sequence as a reference;

[0011] Determine whether there is a segmentation boundary in the current text sequence;

[0012] If it is determined that there is a segmentation boundary in the current text sequence, the selection of the adjacent text sequence is canceled, and a new adjacent text sequence is selected backward according to the preset window length and the segmentation boundary position in the current text sequence, and at the same time, the next text sequence of the preset window length is selected based on the end of the new adjacent text sequence;

[0013] If it is determined that there is no segmentation boundary in the current text sequence, it is determined whether there is a segmentation boundary in an adjacent text sequence, and the next text sequence of the preset window length is selected based on the end of the adjacent text sequence.

[0014] Furthermore, the step of identifying formulas and pictures in the test paper text, converting the formulas and pictures into corresponding characters to obtain processed text, and then dividing the processed text into sentences based on the target symbol includes:

[0015] For the divided sentences, check the length of each sentence and determine whether the length of the sentence is greater than a preset length, wherein the preset length is the maximum sequence length processed by the model or the maximum sequence length supported by the graphics card resources;

[0016] If the length of the sentence is determined to be greater than a preset length, the corresponding sentence is semantically compressed to retain only the core information of the sentence.

[0017] Furthermore, the step of obtaining a database of historical examination papers, randomly extracting parts of historical examination papers in the database, and generating examination paper text according to a preset typesetting format includes:

[0018] Obtain a database of historical examination papers, randomly extract various parts of the historical examination papers in the database, and perform data cleaning on various parts of the historical examination papers in the randomly extracted database;

[0019] Obtain the test questions after data cleaning, and determine whether the test questions are structured questions;

[0020] If it is determined that part of the test questions are structured test questions, a preset strategy is set according to the test question text structure, and the structured test questions are randomly combined to form a test question text;

[0021] According to the typesetting format of the test paper document, the test question text and the non-test question text after data cleaning are combined to generate a first test paper text, wherein the test question text at least includes test questions and answers, and the non-test question text at least includes a test paper title, test instructions, and question type instructions;

[0022] The first test paper text is enhanced to obtain a second test paper text, wherein the enhancement processing at least includes combinations between test questions and between test questions and answers, combinations of test question texts and non-test question texts, and also includes data enhancement processing of a single test question text.

[0023] Furthermore, in the step of identifying formulas and pictures in the test paper text, converting the formulas and pictures into corresponding characters to obtain processed text, and then sentence-segmenting the processed text based on target symbols, when base64 format or accessible links are identified in the test paper text, the identified parts are replaced with preset symbols; when latex format or MathML format is identified in the test paper text, the identified parts are simplified or replaced with Chinese semantic expressions.

[0024] Furthermore, sentences with a start position and an end position are marked as "1", and other sentences are marked as "0".

[0025] Furthermore, the preset window length is dynamically adjusted. When the test paper text to be segmented is segmented into preset progress and each window has at most one segmentation boundary, the number of windows of the initial preset window length and the total number of windows are counted;

[0026] Calculate the ratio of the number of windows with the initial preset window length to the total number of windows, and determine whether the ratio of the number of windows with the preset window length to the total number of windows is greater than a threshold;

[0027] If it is determined that the ratio of the number of windows with the preset window length to the total number of windows is greater than a threshold, the initial preset window length is lengthened.

[0028] Furthermore, the preset window length is dynamically adjusted. When the test paper text to be segmented is segmented, if there are at least two segmentation boundaries in a window, the initial preset window length is shortened.

[0029] A second aspect of an embodiment of the present invention provides a science test paper text segmentation system, which is used to implement the science test paper text segmentation method provided in the first aspect, and the system includes:

[0030] The acquisition module is used to acquire a database of historical test papers, randomly extract various parts of the historical test papers in the database, and generate test paper text according to a preset typesetting format, wherein in the process of generating the test paper text, preset marks are set correspondingly at the starting position and the ending position of each test question;

[0031] A sentence segmentation module is used to identify formulas and pictures in the test paper text, convert the formulas and pictures into corresponding characters to obtain a processed text, and then segment the processed text into sentences based on the target symbol;

[0032] A marking module is used to automatically mark each sentence according to the divided sentences in the processed text, and obtain a sentence list and a corresponding label list of the test paper text, wherein the markings of the sentences with the start position and the end position marks are different from those of other sentences;

[0033] A training module is used to input the sentence list of the test paper text and the data in the corresponding label list into the constructed pointer annotation neural network model for training to obtain a target neural network model;

[0034] The input module is used to input the test paper text to be segmented into the target neural network model and output a segmented test question list.

[0035] A third aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the science test paper text segmentation method provided in the first aspect.

[0036] A fourth aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the science test paper text segmentation method provided in the first aspect is implemented.

[0037] A method and system for segmenting science test paper text provided in an embodiment of the present invention obtains a database of historical test papers, randomly extracts various parts of the historical test papers in the database, and generates test paper text according to a preset typesetting format, wherein, in the process of generating the test paper text, preset marks are set correspondingly at the starting position and the ending position of each test question; formulas and pictures in the test paper text are identified, and the formulas and pictures are converted into corresponding characters to obtain a processed text, and then the processed text is sentence-separated based on a target symbol; according to the sentences that have been segmented in the processed text, each sentence is automatically annotated to obtain a sentence list and a corresponding label list of the test paper text, wherein the annotations of sentences with starting position and ending position marks are different from those of other sentences; the data in the sentence list and the corresponding label list of the test paper text are input into a constructed pointer annotation neural network model for training to obtain a target neural network model; the test paper text to be segmented is input into the target neural network model, and a segmented test question list is output. It should be noted that sentences are encoded by modeling longer contextual relationships, and a hierarchical attention mechanism is built from word level to sentence level to achieve deep semantic understanding. During model training, the setting and interaction of the start position pointer and the end position pointer in the pointer annotation neural network model are used to achieve more accurate test text segmentation. At the same time, during model inference, an adaptive sliding window strategy is adopted to speed up the inference speed and improve the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a method for segmenting science test paper text provided in the first embodiment of the present invention;

[0039] Figure 2 It is a schematic diagram of the structure of the target neural network model;

[0040] Figure 3 A structural block diagram of a science test paper text segmentation system provided in Embodiment 2 of the present invention;

[0041] Figure 4 This is a structural block diagram of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0042] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0043] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0045] Embodiment 1

[0046] See also Figure 1 , Figure 1 A method for segmenting science test paper text provided by the first embodiment of the present invention is shown. The method for segmenting science test paper text specifically includes steps S01 to S05.

[0047] Step S01, obtain a database of historical test papers, randomly extract various parts of the historical test papers in the database, and generate the test paper text according to a preset typesetting format.

[0048] In this embodiment, in the process of generating the test paper text, preset marks are set at the starting position and the ending position of each test question. For example, a preset mark [start] is set at the starting position of each test question, and a preset mark [end] is set at the ending position of each test question.

[0049] Specifically, a database of historical examination papers is obtained, each part of the historical examination papers in the database is randomly extracted, and data cleaning is performed on each part of the historical examination papers in the randomly extracted database;

[0050] Get the test part after data cleaning, and determine whether the test part belongs to a structured test. A structured test can be understood as a test composed of various parts, and each part is stored separately. Taking a multiple-choice question as an example, a structured test can be composed of a stem, various options, an answer, and an analysis, and the stem, various options, an answer, and an analysis are stored separately.

[0051] If it is determined that part of the test questions are structured test questions, a preset strategy is set according to the test question text structure, and the structured test questions are randomly combined to form a test question text;

[0052] According to the typesetting format of the test paper document, the test question text and the non-test question text after data cleaning are combined to generate a first test paper text, wherein the test question text at least includes test questions and answers, and the non-test question text at least includes a test paper title, test instructions, question type instructions, etc.;

[0053] The first test paper text is enhanced to obtain a second test paper text, wherein the enhancement processing at least includes combinations between test questions and between test questions and answers, combinations of test question texts and non-test question texts, and also includes data enhancement processing of a single test question text.

[0054] It can be understood that the data enhancement processing of a single test text can be synonym replacement, random insertion, random exchange, etc. Synonym replacement refers to replacing certain words in the text with their synonyms. For example, replace "fast" with "quickly". This method can increase the variation of the text without changing the original meaning; random insertion refers to inserting words related to the text topic at randomly selected positions in the text to increase the length and complexity of the text, and inserting line breaks at randomly selected positions. When the text is divided into sentences by line breaks, the diversity of sentences can be increased; random exchange refers to randomly selecting two words in the text and exchanging their positions. This method can change the structure of the text, but usually does not change its meaning.

[0055] Step S02, identifying formulas and pictures in the test paper text, converting the formulas and pictures into corresponding characters to obtain processed text, and then dividing the processed text into sentences based on target symbols.

[0056] It should be noted that the test paper text should be divided into sentences first. Since the test paper text is generally very long, it needs to be divided into sentences first. In addition, in actual situations, a sentence often has a line break in the middle. For example, the test paper text recognized by OCR can be divided into sentences based on fixed punctuation marks (such as periods, question marks, etc.) and line breaks.

[0057] In addition, there are generally a large number of formulas and pictures in the test questions text of science papers. The pictures in the test questions text generally exist in the form of base64 format or accessible links and have no specific semantic information, so they can be uniformly processed as fixed symbols, such as "[[Picture]]". The formulas in the test questions text generally exist in the form of latex format or MathML format. However, since their syntax structure forms increase the character length and some characters have no semantic information, it is necessary to simplify these formula formats, such as removing some characters that control typesetting and replacing them with more concise expressions (for example, replacing "\frac{2}{3}" with 2 / 3), or replacing these formulas with Chinese semantic expressions, such as converting "x - y - 4 = 0" to "linear equation with two unknowns", etc. It can be understood that special processing is performed on the formulas and pictures in the test questions text, unifying the picture form information without semantics, simplifying the formula format, highlighting the semantic information implied by the formulas, reducing the information interference of non-critical characters, which helps to encode sentences using longer and more effective context information and improve the more accurate semantic understanding of sentences.

[0058] In some other embodiments of the present invention, for the divided sentences, check the length of each sentence and determine whether the length of the sentence is greater than a preset length, where the preset length is the maximum processing sequence length of the model or the maximum sequence length supported by the graphics card resources.

[0059] If it is determined that the length of the sentence is greater than the preset length, then perform semantic compression on the corresponding sentence to retain only the core information of the sentence. Exemplarily, a method based on a recurrent neural network (RNN) can be used to perform semantic compression on the sentence, mainly using the Seq2Seq model, attention mechanism, gated recurrent unit (GRU), and long short-term memory network (LSTM) in the recurrent neural network. Specifically, an encoder-decoder structure is used. The encoder encodes the input sentence into a vector, and the decoder generates the compressed sentence according to the vector. An attention mechanism is added between the encoder and the decoder to enable the decoder to focus on important words in the input sentence, thereby generating a more accurate compressed sentence. Using GRU or LSTM as the unit of the RNN can better process long sentences and complex semantic information. It can be understood that based on long text processing technology, deeper semantic understanding is achieved by modeling longer context relationships.

[0060] Step S03, according to the sentences already divided in the processed text, automatically annotate each sentence to obtain a sentence list of the test paper text and a corresponding label list, where the annotation of the sentence with start position and end position marks is different from that of other sentences.

[0061] In this embodiment, each sentence is marked based on the divided sentences and the starting and ending position marks of each question in the sample, where sentences containing starting and ending position marks are marked as "1" and other sentences are marked as "0". Finally, a sentence list and a corresponding label list of the test paper text are obtained as training data.

[0062] Step S04, inputting the processed training data into the constructed pointer annotation neural network model for training to obtain the target neural network model.

[0063] The processed training data is the sentence list and corresponding label list of the test paper text. Specifically, the pointer annotation neural network model is built through the Transformer encoder, the sentence-level attention mechanism and the pointer annotation network, wherein the pointer annotation network mainly includes the prediction layer of the start pointer and the end pointer. The sentence-level attention mechanism is combined with the word-level multi-head self-attention mechanism in the Transformer encoder to build a hierarchical attention mechanism from the word level to the sentence level, so as to learn semantic information of different granularities, including the learning of the relationship between sentences and sentences, so as to more accurately capture the semantic information of each sentence text.

[0064] In order to further improve the accuracy of segmentation, sufficient contextual information can be used when encoding sentences, and the sentence vector can be calculated after modeling the dependencies between longer contexts.

[0065] Step S05, input the test paper text to be segmented into the target neural network model, and output the segmented test question list.

[0066] Specifically, for the test paper text to be segmented, before inputting into the target neural network model, it needs to be processed to obtain a sentence list. Then the target neural network model is used to predict the labels of the start position pointer and the end position pointer based on the sentence level. Finally, each question in the test paper text is segmented based on the label sequence of the start position pointer and the end position pointer and combined with some rules.

[0067] See also Figure 2 , is a structural diagram of the target neural network model. For example, after the seven sentences are passed through the Transformer encoder, the sentence-level attention mechanism and the pointer labeling network, the head and tail of each sentence are marked, where the sentences containing the start and end position marks are marked as "1", and the other sentences are marked as "0". Finally, they are divided into test question 1 and test question 2, where test question 1 consists of sentence 1, sentence 2 and sentence 3, and test question 2 consists of sentence 4, sentence 5, sentence 6 and sentence 7.

[0068] Specifically, in the inference process of the target neural network model, an adaptive sliding window strategy is used to perform output pointer sequence inference on the test paper text to be segmented, specifically including:

[0069] Obtaining a preset window length, and determining, according to the preset window length, a current text sequence selected by the preset window length, and an adjacent text sequence selected by the preset window length based on the end of the current text sequence;

[0070] Determine whether there is a segmentation boundary in the current text sequence;

[0071] If it is determined that there is a segmentation boundary in the current text sequence, the selection of the adjacent text sequence is canceled, and a new adjacent text sequence is selected backward according to the preset window length and the segmentation boundary position in the current text sequence. At the same time, the end of the new adjacent text sequence is used as a reference to select the next text sequence of the preset window length;

[0072] If it is determined that there is no segmentation boundary in the current text sequence, it is determined whether there is a segmentation boundary in the adjacent text sequence, and the next text sequence of the preset window length is selected based on the end of the adjacent text sequence.

[0073] It should be noted that the preset window length is dynamically adjusted. When the test paper text to be segmented is segmented to a preset progress, and each window has at most one segmentation boundary, the number of windows of the initial preset window length and the total number of windows are counted. For example, when the segmentation progress of the test paper text to be segmented is 50%, that is, half of the progress is segmented, the total number of windows in the processing process and the number of windows belonging to the initial preset window length are counted. It can be understood that if the length of the window is the initial preset window length, it means that there is no segmentation boundary in the window, that is, no segmentation is performed.

[0074] Calculate the ratio of the number of windows with the initial preset window length to the total number of windows, and determine whether the ratio of the number of windows with the preset window length to the total number of windows is greater than a threshold;

[0075] If the ratio of the number of windows with preset window length to the total number of windows is greater than the threshold, it means that the number of unsegmented windows is large, that is, the segmentation boundary is not found quickly, then the initial preset window length will be lengthened. Specifically, since the overall question setting style of the question setter will remain consistent, that is, the text length of the test question will maintain a certain trend, the lengths of all current windows are counted, and the average is calculated, and the average is determined as the new preset window length for subsequent text segmentation.

[0076] In addition, when the test paper text to be segmented is segmented, there are at least two segmentation boundaries in a window, which means that the window length is too long. In this case, the initial preset window length will be shortened. Specifically, the shortest length target window obtained by dividing the window by the segmentation boundaries is determined, and the length of the target window is used as the new preset window length. At the same time, in the subsequent segmentation process, the above-mentioned dynamic adjustment of the preset window length will continue to be performed, that is, the preset window length may be lengthened subsequently.

[0077] In summary, the method for segmenting the text of science test papers in the above-mentioned embodiments of the present invention obtains a database of historical test papers, randomly extracts various parts of the historical test papers in the database, and generates a test paper text according to a preset typesetting format, wherein, in the process of generating the test paper text, preset marks are set correspondingly at the starting position and the ending position of each test question; the formulas and pictures in the test paper text are identified, and the formulas and pictures are converted into corresponding characters to obtain a processed text, and then the processed text is sentence-separated based on the target symbol; according to the divided sentences in the processed text, each sentence is automatically marked to obtain a sentence list and a corresponding label list of the test paper text, wherein the markings of the sentences with the starting position and the ending position marks are different from those of other sentences; the data in the sentence list and the corresponding label list of the test paper text are input into a constructed pointer annotation neural network model for training to obtain a target neural network model; the test paper text to be segmented is input into the target neural network model, and a segmented test question list is output. Specifically, it should be noted that sentences are encoded by modeling longer contextual relationships, and a hierarchical attention mechanism is built from word level to sentence level to achieve deep semantic understanding. During model training, the setting and interaction of the start position pointer and the end position pointer in the pointer annotation neural network model are used to achieve more accurate test text segmentation. At the same time, during model inference, an adaptive sliding window strategy is adopted to speed up the inference speed and improve the segmentation accuracy.

[0078] Embodiment 2

[0079] See also Figure 3 , Figure 3 1 is a structural block diagram of a science test paper text segmentation system 200 provided in the second embodiment of the present invention. The science test paper text segmentation system 200 includes: an acquisition module 21, a sentence segmentation module 22, a marking module 23, a training module 24 and an input module 25, wherein:

[0080] The acquisition module 21 is used to acquire a database of historical test papers, randomly extract various parts of the historical test papers in the database, and generate test paper text according to a preset typesetting format, wherein in the process of generating the test paper text, preset marks are set at the starting position and the ending position of each test question;

[0081] A sentence segmentation module 22 is used to identify formulas and pictures in the test paper text, convert the formulas and pictures into corresponding characters, obtain a processed text, and then segment the processed text based on the target symbol, wherein when a base64 format or an accessible link is identified in the test paper text, the identified part is replaced with a preset symbol; when a latex format or MathML format is identified in the test paper text, the identified part is simplified or replaced with a Chinese semantic expression;

[0082] The marking module 23 is used to automatically mark each sentence according to the divided sentences in the processed text, and obtain a sentence list and a corresponding label list of the test paper text, wherein the marking of the sentences with the start position and the end position marks is different from that of other sentences, the marking of the sentences with the start position and the end position marks is "1", and the marking of other sentences is "0";

[0083] A training module 24 is used to input the sentence list of the test paper text and the data in the corresponding label list into the constructed pointer annotation neural network model for training to obtain a target neural network model;

[0084] An input module 25 is used to input the test paper text to be segmented into the target neural network model and output a segmented test question list, wherein, in the inference process of the target neural network model, an adaptive sliding window strategy is used to perform output pointer sequence inference on the test paper text to be segmented. Specifically, the preset window length is dynamically adjusted. When the test paper text to be segmented is segmented into a preset progress and each window has at most one segmentation boundary, the number of windows with the initial preset window length and the total number of windows are counted;

[0085] Calculate the ratio of the number of windows with the initial preset window length to the total number of windows, and determine whether the ratio of the number of windows with the preset window length to the total number of windows is greater than a threshold;

[0086] If it is determined that the ratio of the number of windows with the preset window length to the total number of windows is greater than a threshold, the initial preset window length is lengthened;

[0087] When the test paper text to be segmented is segmented, if there are at least two segmentation boundaries in one window, the initial preset window length is shortened.

[0088] Furthermore, in some optional embodiments of the present invention, the acquisition module 21 includes:

[0089] A data cleaning unit is used to obtain a database of historical examination papers, randomly extract various parts of the historical examination papers in the database, and perform data cleaning on various parts of the historical examination papers in the randomly extracted database;

[0090] A first judgment unit is used to obtain the test question part after data cleaning, and judge whether the test question part belongs to a structured test question;

[0091] The first combining unit is used for setting a preset strategy according to the structure of the test question text if it is determined that the test question part belongs to the structured test question, and randomly combining the structured test questions to form the test question text;

[0092] The second combining unit is used to combine the test question text and the non-test question text after data cleaning according to the typesetting format of the test paper document to generate a first test paper text, wherein the test question text at least includes the test question and the answer, and the non-test question text at least includes the test paper title, test description, and question type description;

[0093] An enhancement processing unit is used to perform enhancement processing on the first test paper text to obtain a second test paper text, wherein the enhancement processing at least includes the combination between test questions and between test questions and answers, the combination of test question texts and non-test question texts, and also includes data enhancement processing on a single test question text.

[0094] Furthermore, in some optional embodiments of the present invention, the science test paper text segmentation system 200 further includes:

[0095] A judgment module is used to check the length of each sentence after the division, and judge whether the length of the sentence is greater than a preset length, wherein the preset length is the maximum processing sequence length of the model or the maximum sequence length supported by the graphics card resources;

[0096] The semantic compression module is used to semantically compress the corresponding sentence if it is determined that the length of the sentence is greater than a preset length, so as to retain only the core information of the sentence.

[0097] Embodiment 3

[0098] Another aspect of the present invention provides an electronic device, see Figure 4 , shown is an electronic device in Embodiment 3 of the present invention, including a memory 20, a processor 10, and a computer program 30 stored in the memory and executable on the processor. When the processor 10 executes the computer program 30, the above-mentioned method for segmenting science test paper text is implemented.

[0099] In some embodiments, the processor 10 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 20, such as executing access restriction programs.

[0100] Among them, the memory 20 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 20 can be an internal storage unit of an electronic device, such as a hard disk of the electronic device. In other embodiments, the memory 20 can also be an external storage device of an electronic device, such as a plug-in hard disk equipped on the electronic device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Further, the memory 20 can also include both an internal storage unit of the electronic device and an external storage device. The memory 20 can not only be used to store application software and various types of data of the electronic device, but also can be used to temporarily store data that has been output or is to be output.

[0101] It should be pointed out that Figure 4 The structure shown does not constitute a limitation on the electronic device. In other embodiments, the electronic device may include fewer or more components than those shown in the figure, or combine certain components, or arrange the components differently.

[0102] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for segmenting science test paper text as described above is implemented.

[0103] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0104] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0105] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0106] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0107] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the attached claims.

Claims

1. A method for segmenting science test paper text, characterized in that: The method comprises: Obtain a database of historical test papers, randomly extract parts of the historical test papers in the database, and generate test paper text according to a preset typesetting format, wherein in the process of generating the test paper text, preset marks are set correspondingly at the starting position and the ending position of each test question; Identify formulas and pictures in the test paper text, convert the formulas and pictures into corresponding characters to obtain processed text, and then separate the processed text into sentences based on target symbols; According to the divided sentences in the processed text, each sentence is automatically annotated to obtain a sentence list and a corresponding label list of the test paper text, wherein the annotations of the sentences with the start position and the end position marks are different from those of other sentences, the sentences with the start position and the end position marks are marked as "1", and the other sentences are marked as "0"; Input the sentence list of the test paper text and the data in the corresponding label list into the constructed pointer annotation neural network model for training to obtain the target neural network model; Input the test paper text to be segmented into the target neural network model, output the segmented test question list, and in the reasoning process of the target neural network model, use the adaptive sliding window strategy to perform output pointer sequence reasoning on the test paper text to be segmented, specifically including: Obtaining a preset window length, and determining, according to the preset window length, a current text sequence selected by the preset window length, and an adjacent text sequence selected by the preset window length with the end of the current text sequence as a reference; Determine whether there is a segmentation boundary in the current text sequence; If it is determined that there is a segmentation boundary in the current text sequence, the selection of the adjacent text sequence is canceled, and a new adjacent text sequence is selected backward according to the preset window length and the segmentation boundary position in the current text sequence, and at the same time, the next text sequence of the preset window length is selected based on the end of the new adjacent text sequence; If it is determined that there is no segmentation boundary in the current text sequence, then determining whether there is a segmentation boundary in an adjacent text sequence, and selecting the next text sequence of the preset window length based on the end of the adjacent text sequence; The preset window length is dynamically adjusted. When the test paper text to be segmented is segmented into preset progress and each window has at most one segmentation boundary, the number of windows of the initial preset window length and the total number of windows are counted; Calculate the ratio of the number of windows with the initial preset window length to the total number of windows, and determine whether the ratio of the number of windows with the preset window length to the total number of windows is greater than a threshold; If it is determined that the ratio of the number of windows with the preset window length to the total number of windows is greater than a threshold, the initial preset window length is lengthened; When the test paper text to be segmented is segmented, if there are at least two segmentation boundaries in one window, the initial preset window length is shortened.

2. The method for segmenting science test paper text according to claim 1, characterized in that: The steps of identifying formulas and pictures in the test paper text, converting the formulas and pictures into corresponding characters to obtain processed text, and then dividing the processed text into sentences based on target symbols include: For the divided sentences, check the length of each sentence and determine whether the length of the sentence is greater than a preset length, wherein the preset length is the maximum sequence length processed by the model or the maximum sequence length supported by the graphics card resources; If the length of the sentence is determined to be greater than a preset length, the corresponding sentence is semantically compressed to retain only the core information of the sentence.

3. The method for segmenting science test paper text according to claim 2, characterized in that: The steps of obtaining a database of historical examination papers, randomly extracting parts of the historical examination papers in the database, and generating examination paper text according to a preset typesetting format include: Obtain a database of historical examination papers, randomly extract various parts of the historical examination papers in the database, and perform data cleaning on various parts of the historical examination papers in the randomly extracted database; Obtain the test questions after data cleaning, and determine whether the test questions are structured questions; If it is determined that part of the test questions are structured test questions, a preset strategy is set according to the test question text structure, and the structured test questions are randomly combined to form a test question text; According to the typesetting format of the test paper document, the test question part and the non-test question part after data cleaning are combined to generate a first test paper text, wherein the test question part at least includes test questions and answers, and the non-test question part at least includes a test paper title, test instructions, and question type instructions; The first test paper text is enhanced to obtain a second test paper text, wherein the enhancement processing at least includes combinations between test questions and between test questions and answers, combinations of test question parts and non-test question parts, and also includes data enhancement processing of a single test question text.

4. The method for segmenting science test paper text according to claim 3, characterized in that: In the step of identifying formulas and pictures in the test paper text, converting the formulas and pictures into corresponding characters to obtain processed text, and then sentence-segmenting the processed text based on target symbols, when base64 format or accessible links are identified in the test paper text, the identified parts are replaced with preset symbols; when latex format or MathML format is identified in the test paper text, the identified parts are simplified or replaced with Chinese semantic expressions.

5. A science test paper text segmentation system, characterized in that: Used to implement the method for segmenting science test paper text as claimed in any one of claims 1 to 4, the system comprises: The acquisition module is used to acquire a database of historical test papers, randomly extract various parts of the historical test papers in the database, and generate test paper text according to a preset typesetting format, wherein in the process of generating the test paper text, preset marks are set correspondingly at the starting position and the ending position of each test question; A sentence segmentation module is used to identify formulas and pictures in the test paper text, convert the formulas and pictures into corresponding characters to obtain a processed text, and then segment the processed text into sentences based on the target symbol; A marking module is used to automatically mark each sentence according to the divided sentences in the processed text, and obtain a sentence list and a corresponding label list of the test paper text, wherein the markings of the sentences with the start position and the end position marks are different from those of other sentences; A training module is used to input the sentence list of the test paper text and the data in the corresponding label list into the constructed pointer annotation neural network model for training to obtain a target neural network model; The input module is used to input the test paper text to be segmented into the target neural network model and output a segmented test question list.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for segmenting science test paper text as described in any one of claims 1 to 4 is implemented.

7. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for segmenting science test paper text as claimed in any one of claims 1 to 4 is implemented.

Images