Text difficulty quantification method and system based on character continuation
Through the word -based continuation method, the word continuity index in Chinese text is extract and calculate the difficulty of neglecting text combinations in existing technologies, and achieve more accurate and efficient text difficulty.
Patent Information
- Application Number
- CN202411385536.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-01-24
AI Technical Summary
When measuring the difficulty of Chinese text, the existing technology mainly relies on multi-dimensional calculations of static language units, and ignores the measurement of text combination difficulty, resulting in insufficient quantification accuracy and efficiency.
A text difficulty quantification method based on word continuation is used to construct a graded corpus of student texts by obtaining student texts of different difficulty levels. The word continuation index is extracted and calculated to form a word continuation corpus. The relative distance method is used to calculate the difficulty level of the text to be quantified.
The accuracy and efficiency of the difficulty of student text have been improved. It can comprehensively measure the static and combined difficulty of Chinese text. It does not require complex language characteristics, less measurement indicators, convenient methods, and high calculation efficiency.
Smart Images
Figure CN119357390A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of text difficulty measurement, and in particular to a text difficulty quantification method and system based on word continuation. Background Art
[0002] In the prior art, the commonly used methods for evaluating and measuring text difficulty include formula methods and text classification methods based on complex feature calculations. In terms of formula methods, well-known readability formulas developed abroad include Flesch-Kincaid, Fry, FOG, SMOG, Dale-Chall, FORCAST, etc.; domestic scholars have developed different readability formulas for different types of texts, such as traditional Chinese texts, Chinese teaching texts for foreigners, and Chinese textbooks. Whether abroad or at home, the language indicators included in the calculation of readability formulas mainly include the number of words, the proportion of difficult words, the number of function words, the number of sentences, the length of sentences, the proportion of difficult characters, etc., all of which measure text difficulty through various static language units.
[0003] The above studies construct language feature systems from different levels to measure the difficulty of Chinese texts. The language features used are complex, but these features basically belong to the multi-dimensional calculation of the static difficulty of the text, and ignore the measurement of the combination difficulty of the text. Summary of the invention
[0004] The purpose of the present invention is to provide a method and system for quantifying text difficulty based on word continuation, which method and system are conducive to improving the accuracy and efficiency of quantifying the difficulty of student texts.
[0005] In order to achieve the above object, the technical solution adopted by the present invention is: a text difficulty quantification method based on word continuation, comprising the following steps:
[0006] Step S1, obtaining student texts of different difficulty levels, forming training text sets of different difficulty levels, and then constructing a graded corpus of student texts;
[0007] Step S2, taking the string of words that appear in sequence in the text, i.e., word continuation, as the evaluation object of text difficulty; for all training texts in the student text graded corpus, extracting the word continuation in the text, calculating the word continuation index of all the words appearing therein, and forming a word continuation corpus; based on the word continuation corpus, calculating the statistical indicators related to the word continuation index of the training text sets of different difficulty levels;
[0008] Step S3: For the text to be quantified, based on the word continuation index-related statistical indicators of the word continuation corpus and the training text sets of different difficulty levels, the relative distance method is used to calculate the distance between the text to be quantified and the training text sets of different difficulty levels. The difficulty level corresponding to the shortest distance is the difficulty level of the text to be quantified.
[0009] Furthermore, in step S1, student texts of different difficulty levels are obtained from publicly available real student texts rather than online text corpora. The obtained student texts have a graded attribute and can be labeled and classified according to the difficulty level of the students' grade or stage of study.
[0010] Furthermore, the word continuation is obtained through an N-gram model in units of characters; the word continuation includes static language units of the text such as words, language blocks, word parts and block parts, which can measure the static difficulty of the text, and also includes words and blocks, which can reflect the combination difficulty of the text, and is a comprehensive measure of the static difficulty and combination difficulty of the text.
[0011] Furthermore, the character continuation index of a character refers to the number of types of two-character continuations, three-character continuations, or more character continuations that contain the character in all texts in which the character appears.
[0012] Furthermore, the specific implementation method of step S2 is:
[0013] 1) Perform word continuation segmentation on all training texts in the student text classification corpus;
[0014] 2) For each character that appears in all training texts, the number of types of character continuations corresponding to the character is calculated, thereby obtaining the character continuation index of all characters and forming a character continuation corpus;
[0015] 3) For each set of training texts at each difficulty level, calculate the average word continuation index and the standard deviation of the word continuation index of each text;
[0016] 4) Calculate the average value of the average word continuation index of all texts in the training text set of each difficulty level and the average value of the standard deviation of the word continuation index.
[0017] Furthermore, the calculation formula for the average word continuation index of a single text is:
[0018]
[0019] in, is the average word continuation index of the jth text in the i-th difficulty level training text set; x i,j,k is the word continuation index of the kth word in the jth text of the i-th difficulty level training text set, obtained from the word continuation corpus; t is the number of words in the text;
[0020] The calculation formula of the standard deviation of the word continuation index of a single text is:
[0021]
[0022] Among them, y i,jis the standard deviation of the word continuation index of the jth text in the i-th difficulty level training text set;
[0023] The calculation formula for the average value of the average word continuation index of all texts in a training text set is:
[0024]
[0025] in, is the average of the average word continuation indexes of all texts in the training text set of the i-th difficulty level;
[0026] The calculation formula for the average value of the standard deviation of the word continuation index of all texts in a training text set is:
[0027]
[0028] in, is the average value of the standard deviation of the word continuation index of all texts in the training text set of the i-th difficulty level.
[0029] Furthermore, in step S3, the calculation formula of the distance ρ between a text to be quantified and a difficulty level training text set is:
[0030]
[0031] Among them, a is the average word continuation index of the text to be quantified, is the average value of the average word continuation index of all texts in the training text set of the i-th difficulty level, b is the standard deviation of the word continuation index of the text to be quantified, is the average value of the standard deviation of the word continuation index of all texts in the training text set of the i-th difficulty level.
[0032] Furthermore, the method can be applied to realize automatic evaluation of four difficulty level indicators of primary school students' texts, namely grade difficulty, adjacent grade difficulty, three-stage difficulty, and two-stage difficulty.
[0033] The present invention also provides a text difficulty quantification system based on word continuation, comprising a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the above method steps can be implemented.
[0034] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a text difficulty quantification method and system based on word continuation, which takes word continuation as the evaluation object of text difficulty, shifts the measurement perspective of Chinese text difficulty from the previous static unit difficulty to the continuation combination difficulty of text expression elements, and uses the new method to comprehensively measure the difficulty of Chinese text, which can not only improve the accuracy of quantification of students' text difficulty, but also the measurement process does not require complex language features, has fewer measurement indicators, is convenient in measurement method, has high calculation efficiency, and has strong practicality and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a flow chart of the method implementation of the embodiment of the present invention. DETAILED DESCRIPTION
[0036] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0037] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0038] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0039] like Figure 1 As shown, this embodiment provides a text difficulty quantification method based on word continuation, comprising the following steps:
[0040] Step S1: Obtain student texts of different difficulty levels, form training text sets of different difficulty levels, and then construct a graded corpus of student texts.
[0041] In this embodiment, student texts of different difficulty levels are obtained from publicly published real student texts rather than online text corpora to ensure the reliability of subsequent quantitative judgments. The obtained student texts have a graded attribute, and the difficulty level of the texts can be marked and classified according to the grade or stage of study of the students.
[0042] Step S2, taking the string of words that appear in sequence in the text, i.e., word continuation, as the object of text difficulty assessment; for all training texts in the student text graded corpus, extracting the word continuation in the text, calculating the word continuation index of all the words appearing therein, and forming a word continuation corpus; based on the word continuation corpus, calculating the statistical indicators related to the word continuation index of the training text sets of different difficulty levels.
[0043] Among them, word continuation can be two-word continuation, three-word continuation, etc. The more word continuation types a certain character has, the more widely known it is, and the lower its difficulty. Word continuation is obtained through the N-gram model in units of characters. Word continuation includes static language units of text such as words, language blocks, word parts and block parts, which can measure the static difficulty of the text. It also includes words and blocks, which can reflect the combination difficulty of the text. It is a comprehensive measure of the static difficulty and combination difficulty of the text.
[0044] The character continuation index of a certain character refers to the number of types of two-character continuation, three-character continuation, or more-character continuation, etc., containing the character in all texts in which the character appears.
[0045] In this embodiment, the specific implementation method of step S2 is:
[0046] 1) Perform word continuation segmentation on all training texts in the student text grading corpus.
[0047] 2) For each character that appears in all training texts, the number of corresponding character continuation types is calculated, thereby obtaining the character continuation index of all characters and forming a character continuation corpus.
[0048] 3) For the training text sets of each difficulty level, calculate the average word continuation index and the standard deviation of the word continuation index of each text.
[0049] The calculation formula for the average word continuation index of a single text is:
[0050]
[0051] in, is the average word continuation index of the jth text in the i-th difficulty level training text set; x i,j,k is the word continuation index of the kth word in the jth text in the i-th difficulty level training text set, obtained from the word continuation corpus; t is the number of words in the text.
[0052] The calculation formula of the standard deviation of the word continuation index of a single text is:
[0053]
[0054] Among them, y i,jis the standard deviation of the word continuation index of the jth text in the i-th difficulty level training text set.
[0055] 4) Calculate the average value of the average word continuation index of all texts in the training text set of each difficulty level and the average value of the standard deviation of the word continuation index.
[0056] The calculation formula for the average value of the average word continuation index of all texts in a training text set is:
[0057]
[0058] in, is the average of the average word continuation indexes of all texts in the training text set of the i-th difficulty level;
[0059] The calculation formula for the average value of the standard deviation of the word continuation index of all texts in a training text set is:
[0060]
[0061] in, is the average value of the standard deviation of the word continuation index of all texts in the training text set of the i-th difficulty level.
[0062] Step S3: For the text to be quantified, based on the word continuation index-related statistical indicators of the word continuation corpus and the training text sets of different difficulty levels, the relative distance method is used to calculate the distance between the text to be quantified and the training text sets of different difficulty levels. The difficulty level corresponding to the shortest distance is the difficulty level of the text to be quantified.
[0063] The calculation formula for the distance ρ between a text to be quantified and a set of training texts of a certain difficulty level is:
[0064]
[0065] Among them, a is the average word continuation index of the text to be quantified, is the average value of the average word continuation index of all texts in the training text set of the i-th difficulty level, b is the standard deviation of the word continuation index of the text to be quantified, is the average value of the standard deviation of the word continuation index of all texts in the training text set of the i-th difficulty level.
[0066] After calculating the distance between the text to be quantified and each difficulty level training text set by the above formula, it is determined which difficulty level training text set the text to be quantified is closest to, and the difficulty level is the difficulty level of the text to be quantified.
[0067] This embodiment also provides a text difficulty quantification system based on word continuation, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the above method steps can be implemented.
[0068] The text difficulty quantification method based on word continuation proposed in the present invention can be applied to realize automatic evaluation of four difficulty level indicators of primary school students' texts, namely grade difficulty, adjacent grade difficulty, three-stage difficulty, and two-stage difficulty.
[0069] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0070] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0071] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0072] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.
[0073] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.
Claims
1. A text difficulty quantification method based on word continuation, characterized in that: The following steps are involved: Step S1, obtaining student texts of different difficulty levels, forming training text sets of different difficulty levels, and then constructing a graded corpus of student texts; Step S2, taking the string of words that appear in sequence in the text, i.e., word continuation, as the evaluation object of text difficulty; for all training texts in the student text classification corpus, extract the word continuation in the text, calculate the word continuation index of all words appearing therein, and form a word continuation corpus; Based on the word continuation corpus, calculate the word continuation index related statistical indicators of the training text sets of different difficulty levels; Step S3: For the text to be quantified, based on the word continuation index-related statistical indicators of the word continuation corpus and the training text sets of different difficulty levels, the relative distance method is used to calculate the distance between the text to be quantified and the training text sets of different difficulty levels. The difficulty level corresponding to the shortest distance is the difficulty level of the text to be quantified.
2. A text difficulty quantification method based on word continuation according to claim 1, characterized in that: In step S1, student texts of different difficulty levels are obtained from publicly available real student texts rather than online text corpora. The obtained student texts have a graded attribute and can be labeled and classified according to the difficulty level of the students' grade or stage of study.
3. A text difficulty quantification method based on word continuation according to claim 1, characterized in that: The word continuation is obtained through the N-gram model in units of words; the word continuation includes static language units of the text such as words, language blocks, word parts and block parts, which can measure the static difficulty of the text, and also includes words and blocks, which can reflect the combination difficulty of the text. It is a comprehensive measure of the static difficulty and combination difficulty of the text.
4. A text difficulty quantification method based on word continuation according to claim 1, characterized in that: The character continuation index of a character refers to the number of types of two-character continuation, three-character continuation or more character continuation containing the character in all texts in which the character appears.
5. A text difficulty quantification method based on word continuation according to claim 1, characterized in that: The specific implementation method of step S2 is: 1) Perform word continuation segmentation on all training texts in the student text classification corpus; 2) For each character that appears in all training texts, the number of types of character continuations corresponding to the character is calculated, thereby obtaining the character continuation index of all characters and forming a character continuation corpus; 3) For each set of training texts at each difficulty level, calculate the average word continuation index and the standard deviation of the word continuation index of each text; 4) Calculate the average value of the average word continuation index of all texts in the training text set of each difficulty level and the average value of the standard deviation of the word continuation index.
6. A text difficulty quantification method based on word continuation according to claim 5, characterized in that: The calculation formula for the average word continuation index of a single text is: in, is the average word continuation index of the jth text in the i-th difficulty level training text set; x i,j,k is the word continuation index of the kth word in the jth text of the i-th difficulty level training text set, obtained from the word continuation corpus; t is the number of words in the text; The calculation formula of the standard deviation of the word continuation index of a single text is: Among them, y i,j is the standard deviation of the word continuation index of the jth text in the i-th difficulty level training text set; The calculation formula for the average value of the average word continuation index of all texts in a training text set is: in, is the average of the average word continuation indexes of all texts in the training text set of the i-th difficulty level; The calculation formula for the average value of the standard deviation of the word continuation index of all texts in a training text set is: in, is the average value of the standard deviation of the word continuation index of all texts in the training text set of the i-th difficulty level.
7. A text difficulty quantification method based on word continuation according to claim 1, characterized in that: In step S3, the calculation formula of the distance ρ between a text to be quantified and a difficulty level training text set is: Among them, a is the average word continuation index of the text to be quantified, is the average value of the average word continuation index of all texts in the training text set of the i-th difficulty level, b is the standard deviation of the word continuation index of the text to be quantified, is the average value of the standard deviation of the word continuation index of all texts in the training text set of the i-th difficulty level.
8. A text difficulty quantification method based on word continuation according to claim 1, characterized in that: This method can be applied to realize the automatic evaluation of four difficulty level indicators of primary school students' texts: grade difficulty, adjacent grade difficulty, three-stage difficulty, and two-stage difficulty.
9. A text difficulty quantification system based on word continuation, characterized in that: The method comprises a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the method steps as described in any one of claims 1 to 8 can be implemented.