A Chinese text readability grading method based on multi-level language feature fusion

Abstract

This invention discloses a Chinese text readability classification method based on multi-level language feature fusion. The method includes: a language level feature extraction module extracting level features of Chinese characters, vocabulary, and grammar; a multi-level feature fusion module deeply fusing language level features with deep semantic features, and further fusing language level features at the model embedding layer and self-attention layer. Compared with existing technologies, this invention outperforms all baseline models in the Chinese text readability classification task. In terms of language feature extraction, it can more fully preserve text readability information. Based on the traditional BERT-based text readability classification model, a multi-level language feature fusion method is designed to fuse the language features of Chinese characters, vocabulary, and grammar with the model's embedding layer and self-attention layer. The method is simple and practical, achieving an accuracy of 94.2% on the test set, thus contributing to Chinese education.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a method for multidimensional readability analysis of Chinese text and a method for classifying the readability of Chinese text based on the fusion of multi-level language features. Technical Background

[0002] The international Chinese language education is developing rapidly, with an increasing number of learners venturing into the field. This diverse group of learners encompasses different countries, cultures, and linguistic contexts, with varying levels of Chinese proficiency, academic backgrounds, and learning objectives. This makes the readable classification of Chinese texts increasingly important. In the context of international Chinese language education, learners' needs and backgrounds differ greatly. Some learners may be beginners who hope to start with simple and easy-to-understand Chinese texts to ensure a smooth reading process; others may already have a certain level of Chinese proficiency and seek a higher level of language challenge, choosing reading materials that are challenging yet not overly difficult. By scientifically and rationally classifying the difficulty levels of Chinese texts, learners can more effectively select materials suitable for their own level, achieving a gradual and efficient Chinese learning process. This plays a vital role in improving the Chinese reading proficiency of international learners and promoting intercultural communication. Meanwhile, the readability grading of Chinese texts not only serves individual learners but also provides international Chinese teachers and online education platforms with a more scientific and personalized means of managing teaching resources. Through graded Chinese texts, teachers and platforms can better organize and optimize teaching content, providing learning paths that better meet learners' needs, thereby improving the quality and effectiveness of online Chinese education. Furthermore, the Chinese text readability grading task has had a profound impact on the education field. Textbook writers can better create teaching materials based on subject characteristics and student levels, curriculum planners can more scientifically design phased learning objectives, and automatic essay scoring systems can more accurately match students' writing levels. Such tasks not only improve teaching effectiveness but also provide strong support for the sustainable development of Chinese education.

[0003] However, existing methods for classifying Chinese text readability suffer from several drawbacks: limited text feature extraction, inability to comprehensively represent text readability information, insufficient use of deep semantic features, and superficial fusion methods between linguistic and deep semantic features. These shortcomings significantly hinder the accuracy and reliability of Chinese text readability classification. Specifically, the problems with existing technologies can be summarized in the following three aspects:

[0004] 1) The text feature extraction is relatively simplistic and fails to comprehensively represent the readability information of the text. The Chinese text readability grading task requires extracted features to include as much information as possible that helps differentiate text difficulty. Since the task of measuring Chinese text readability is closely related to linguistic knowledge, linguistic features such as text length, average stroke count, and the proportion of difficult words are used extensively. However, information on the levels of Chinese characters, vocabulary, and grammar in the language syllabus is used less frequently, and its use often remains limited to simply representing the proportion of each difficulty category in the text, resulting in an incomplete utilization of level information, especially grammatical level information. For example, the publication "CN115859962A" obtains character features, word features, sentence features, and discourse features of the text to be evaluated in the feature acquisition module, and then constructs a readability formula for regression analysis. Its feature settings do not consider the level information of Chinese characters, vocabulary, and grammar, and its use of the level syllabus is limited to extracting the proportion of complex words and the density of advanced grammatical points.

[0005] 2) Deep semantic features of text are rarely used, and their use is mostly limited to directly using their representation vectors while ignoring the deep model structure. Currently, with the rise of pre-trained models such as BERT, some works have extracted the first and last vectors of the model's output layer as the semantic representation vectors of the text. However, since the pre-training task of the BERT model differs from the current text classification task, directly taking the vector representations from the pre-training model may not be suitable for direct application in the current scenario. Moreover, existing works on the use of the deep learning model BERT are limited to using its sentence vector representations, without considering the combination of BERT's internal network structure and language features. For example, the publication "CN115630155A" defines an indicator system from three dimensions—text cognition, text sentiment, and text function—in the indicator construction module, and classifies the difficulty of Chinese children's books based on the principle of faceted grouping classification. However, its features rely on the graded character list, graded word list, and multi-dimensional sentiment word list, which are relatively shallow features and do not use deep semantic features.

[0006] 3) The fusion of linguistic features and deep semantic features is relatively superficial and cannot fully leverage the complementary roles of both in representing text readability. Existing methods often employ hierarchical pre-fusion or hierarchical post-fusion approaches. The former maps each feature to the same space and adds them together to form a total feature, which is then used by a classifier model to predict the level of text readability. The latter constructs separate classifiers for each feature to predict the difficulty level of the text, and then calculates the probability distribution of the difficulty level under different features to obtain the final difficulty level. However, the fusion strategies used in current research are relatively simple, often using vector concatenation or mapping methods, with limited interaction between features, failing to maximize the complementary role of linguistic features and deep semantic features. For example, the publication "CN113934850A" constructed a candidate set of text features, including characters and words, for fitting readability formula parameters and training machine learning models, but it has not yet addressed the fusion of linguistic features and deep semantic features. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a Chinese text readability classification method based on multi-level language feature fusion. This method employs a hierarchical feature extraction approach for Chinese characters, vocabulary, and language, deeply fusing language hierarchical features with deep semantic features. Using a trained Chinese text readability classification model, it predicts the difficulty level of the input text. Based on the traditional BERT text readability classification model, this method designs a multi-level language feature fusion approach, considering the interaction between different language features and network layer structures, fusing language features of Chinese characters, vocabulary, and grammar with the model's embedding and self-attention layers. In terms of language feature extraction, a feature storage format combining feature vectors and feature matrices is designed to more fully retain text readability information. Experimental results show that this method outperforms all baseline models on the Chinese text readability classification task, achieving an accuracy of 94.2% on the test set, thus contributing to Chinese education.

[0008] The specific technical solution to achieve the purpose of this invention is: a Chinese text readability classification method based on multi-level linguistic feature fusion. Its characteristics include: employing a Chinese text readability classification model based on multi-level linguistic feature fusion; addressing the problem that text feature extraction is relatively singular and cannot comprehensively represent the readability information of the text; extracting character, vocabulary, and grammatical features from the text; and designing targeted multi-dimensional feature representation methods. Addressing the problem that deep semantic features are used relatively superficially, based on the traditional BERT-based text readability classification model, linguistic features are treated as additional knowledge and fused into the model. Addressing the problem that the fusion of linguistic features and deep semantic features is relatively superficial, a multi-level linguistic feature fusion (MLF) method is designed to fuse the features of Chinese characters, vocabulary, and grammar into the model's embedding layer and self-attention layer, which can better integrate linguistic features and utilize linguistic information. The method specifically includes the following steps:

[0009] Step 1: Construct a traditional text readability classification model based on BERT

[0010] The goal of text classification is to train a classifier that maps input text sequences to category labels. In text readability grading, this translates to mapping text sequences to difficulty levels. The input text set is defined as follows: The collection contains N texts, where T is the i-th text. Each text consists of a character sequence, T = (t, t, ..., t), where t is the l-th character of the text. The output text category is y ∈ C, where C is the set of difficulty levels. The specific steps are as follows:

[0011] 1-1. First, input the article T into the embedding layer. In this layer, the input text will be transformed into an l×d dimension matrix through token embeddings, segment embeddings, and position embeddings, where d is the length of the character embedding vector.

[0012] 1-2. Each Transformer encoder layer consists of two sub-layers: a self-attention layer and a feedforward layer. The output of the k-th layer... It can be expressed by the following equation (e):

[0013]

[0014] Among them, A () =Q () K ()T For the attention score of the k-th layer, This is the query matrix for the k-th level. Let k be the key matrix. Let be the value matrix for the k-th layer. For ease of description, the above only shows the calculation of a single attention head; the actual calculation involves concatenating multiple attention heads. d represents the number of attention heads in the k-th layer. This is the embedding matrix of the input text sequence.

[0015] 1-3. To achieve the classification task, the hidden representation vector corresponding to "[CLS]" in the last layer is input into a fully connected layer. Assuming a class set of size |C| is used, the output of the classifier can be expressed by the following equation (f):

[0016]

[0017] Among them, W (fc) It is a |C|×d weight matrix, b (fc) It is a bias vector of length |C|. This is the hidden representation vector of the last layer "[CLS]" of the input sequence. This completes the text classification process based on BERT.

[0018] Step 2: Extracting Language Level Features

[0019] The "three grades and nine levels" syllabus for Chinese characters, vocabulary, and grammar (International Chinese Language Education Chinese Proficiency Level Standards) is used to analyze the difficulty of the Chinese characters, vocabulary, and grammar contained in the text. Specifically, the extracted features include three language level features: Chinese characters, vocabulary, and grammar. Each feature corresponds to a feature vector and a feature matrix (V, M). The extraction method for each feature is as follows:

[0020] 2-1. Character Hierarchical Feature Extraction

[0021] Construct the eigenvector V cThe specific method is as follows: traverse the text. If a non-Chinese character is encountered, directly assign the vector value at that position to "0". If a Chinese character is encountered, find its corresponding difficulty level according to the Chinese character difficulty level outline provided in the *International Chinese Language Education Chinese Proficiency Level Standards*, and assign the level number to the vector value at that position. If the Chinese character cannot be found in the Chinese character difficulty level outline provided in the *International Chinese Language Education Chinese Proficiency Level Standards*, assign the vector value at that position to "0". In this way, the input article is converted into an integer vector of the same length as the input article. The integer values ​​of each dimension of this vector range from 0 to 7. This yields the Chinese character level feature vector V of the article. c = (c,...,c,...,c), where c is the level of the i-th Chinese character and l is the length of the article.

[0022] For the Chinese character level feature matrix M c Then, the feature vector of the Chinese character level is used as the value on the diagonal of the diagonal matrix. This matrix is ​​a diagonal matrix with a size of l×l.

[0023] 2-2. Lexical level feature extraction

[0024] After performing word segmentation on the article, this invention matches each word with the difficulty level in the outline. Finally, the vector value of the position of the unmatched character in the article is assigned "0", and the vector value of the position of the other words matched in the vocabulary outline is assigned the level number.

[0025] This invention assigns a value of "0" to non-Chinese characters and Chinese characters and words that do not match in the outline. This operation reduces the model's attention to non-Chinese characters, making it focus more on Chinese characters. Meanwhile, some words among the unmatched words (e.g., "words beyond the outline") may be helpful for the model in judging the readability of the article, but the definition of such words is relatively vague and their number is small; therefore, this invention does not treat them specially.

[0026] Similar to the output vector of the Chinese character level features, this invention transforms the input article into an integer vector of the same length as the input article, where the integer value of each dimension ranges from 0 to 7. This is used to construct the vocabulary level feature vector V. w The vector has a length of l. For the vocabulary level feature matrix M... w If a word is positioned [i:j] in the vector after word segmentation, and its corresponding difficulty level is d, then the corresponding consecutive positions [i:j,i:j] in the vocabulary level feature matrix are assigned the value d. This matrix is ​​a symmetric matrix with a size of l×l.

[0027] 2-3. Syntax Level Feature Extraction

[0028] The grammar in an article determines the arrangement of Chinese characters and words, thus significantly affecting its readability. Therefore, this invention constructs grammatical features (V). g M g ).

[0029] Using regular expressions, this invention matches grammar points in an article. Based on the grammar difficulty level outline provided by the "International Chinese Language Education Chinese Proficiency Level Standards", it finds the corresponding difficulty level for the matched grammar points and assigns the vector value of the non-continuous grammar marker of that grammar point in the article as the level number.

[0030] After the above operations, the input article will be converted into an integer vector of the same length as the output, where the integer value of each dimension of the vector ranges from 0 to 7, thus constructing the grammar level feature vector V. g The length of this vector is l.

[0031] For the grammar level feature matrix M g To fully utilize the characteristics of non-continuous grammar, this invention uses a quasi-adjacency matrix to represent grammatical features, as detailed in the appendix. Figure 1 (c). Specifically, a certain syntax is positioned in the text as [i1:j1]...[i k :j k Let k be the number of non-contiguous grammatical markers contained in the grammar point, and d be the corresponding difficulty level. Then, the corresponding continuous positions in the grammar level feature matrix are [i1:j1,i1:j1]...[i k :j k i k :j k The matrix is ​​assigned the value d. This matrix is ​​a symmetric matrix with a size of l×l. See the appendix for details. Figure 1 (c)

[0032] Step 3: A text readability ranking model based on multi-level language feature fusion and BERT

[0033] In the internal network structure of the traditional BERT model, two fusion strategies are used to deeply fuse the language-level features of "Chinese characters, vocabulary, and grammar" in the article at the embedding layer and the self-attention layer in the Transformer encoder layer, respectively. Appropriate fusion strategies are selected for the "Chinese characters, vocabulary" and "grammar" features respectively to promote deep fusion between features and fully leverage their complementary effects.

[0034] 3-1. Embedding layer incorporates language hierarchy features

[0035] In the BERT model, the embedding layer converts the input text sequence into a fixed-length vector representation to facilitate subsequent model processing; each subsequent layer is influenced by the output of this layer. Based on the above findings, this invention incorporates the language level features extracted in the previous section, based on the "International Chinese Language Education Chinese Proficiency Level Standards," into the BERT embedding layer. All subsequent layers can receive the language level feature information and propagate this information layer by layer using a self-attention mechanism. Ultimately, this assists in predicting the difficulty level of the text.

[0036] In addition to the three existing embedding parts in the original BERT model's embedding layer, language-level feature embedding parts are added, namely character-level embeddings, word-level embeddings, and grammar-level embeddings.

[0037] The embedded parts for Chinese characters, vocabulary, and grammar are calculated using the following formula (b):

[0038] t m =Embedding(V m ), m∈{c,w,g} (b).

[0039] Among them, V m For the hierarchical feature vector, t m Embedded as hierarchical features.

[0040] Subsequently, the language-level feature embeddings (including character-level embeddings, lexical-level embeddings, and grammatical-level embeddings) are added to the original three embeddings of BERT to obtain the embedding representation of the article as shown in equation (a) below.

[0041]

[0042] t j =t token +t seg +t pos +t char +t word +t gram (a).

[0043] Among them, t j Let t be the embedding vector representation of the j-th character in the article. j =t token +t seg +t pos +t char +t word +tgram ; where t token t seg t pos t char t word t gram These are word embedding vectors, sentence segmentation embedding vectors, positional embedding vectors, Chinese character-level feature embedding vectors, lexical-level feature embedding vectors, and grammatical-level feature embedding vectors, respectively.

[0044] This completes the operation of integrating language level features into the embedding layer. The output of this layer will contain the language level features of the input article, which will propagate along with the forward propagation of the BERT layer and assist in the prediction of the article difficulty level in the final output layer.

[0045] This fusion method has the advantages of being simple, easy to implement, convenient and fast; however, as information is passed layer by layer, the language level feature information integrated into the embedding layer will be gradually diluted. After the self-attention mechanism of 12 Transformer layers is used for calculation, the language level feature information of the output layer will be very sparse.

[0046] 3-2. Integrating Language Hierarchy Features into the Self-Attention Layer

[0047] To address the issue of information attenuation caused by incorporating language-level features into the embedding layer, this invention adds a feature fusion point. Specifically, language-level feature information is incorporated into the self-attention layer within the Transformer layer of the BERT model. This operation, on the one hand, effectively avoids the problem of information attenuation by adding language-level feature information to all 12 Transformer layers; on the other hand, the attention score matrix characterizes the interrelationships between characters, facilitating the full capture of relationships between grammatical markers.

[0048] Based on the BERT model's calculation of the attention score, i.e., formula (4), this invention incorporates the language level feature matrix into the attention A calculated by the following formula (c):

[0049] A = Q * K T +M c +M w +M g (c)

[0050] Among them, M c M is the feature matrix of Chinese characters; w M is the vocabulary feature matrix; g This is the grammatical feature matrix. As the formula shows, the higher the difficulty of a language feature formed by two characters, the greater its corresponding attention value.

[0051] Based on the above method, this invention integrates language level feature information into the self-attention score. As information is passed between each layer of the Transformer, this language level feature information is passed layer by layer. Compared to methods that directly integrate level information into the embedding layer, this method can, to some extent, avoid the problem of information loss layer by layer. In the most extreme case, language level information can be added to each layer, thereby strengthening the influence of language level features on the model. Furthermore, for non-continuous grammatical features, feature fusion at the self-attention layer can effectively represent and capture the relationships between characters.

[0052] 3-3. Loss Function Setting

[0053] This invention uses the cross-entropy loss function. The article set T contains N samples, and the number of categories is |C|. The cross-entropy loss function can be expressed by the following equation (d):

[0054]

[0055] Where θ is the model parameter, y i,j This represents the true probability that the i-th sample belongs to the j-th class. This represents the probability that the model predicts the sample belongs to the j-th category.

[0056] The training objective is to minimize the value of the cross-entropy loss function, which is expressed by the following equation (g):

[0057] θ * =argmin θ J(θ) (g).

[0058] Compared with the prior art, the present invention has the following advantages:

[0059] 1) The language features extracted and their storage method in this invention can effectively represent the multidimensional readability of text. This invention uses the difficulty outline of Chinese characters, vocabulary, and grammar provided by the "International Chinese Language Education Chinese Proficiency Level Standards" to annotate the difficulty of Chinese characters, vocabulary, and grammar in the text. It uses vector and matrix formats for storage. Vector storage can accurately annotate the difficulty information of Chinese characters and vocabulary in the text; while matrix format is beneficial for annotating non-continuous grammatical information used in the text. Comprehensive consideration of text features and reasonable design of storage methods can fully preserve the readability representation of the text.

[0060] 2) The text readability classification model designed in this invention, based on multi-level language feature fusion and BERT, can improve the accuracy of text readability classification. This invention deeply fuses the language level features of "Chinese characters, vocabulary, and grammar" in the embedding layer and the self-attention layer in the Transformer encoder layer. Appropriate fusion strategies are selected for "Chinese characters, vocabulary," and "grammar" features respectively to promote deep fusion between features and fully leverage their complementary effects. This invention is trained and tested using a Chinese text readability classification dataset. On the test set, it achieves an accuracy of 94.24%, surpassing all other baseline models. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the training and prediction process of the present invention;

[0062] Figure 2 This is a schematic diagram of the model architecture of the present invention;

[0063] Figure 3 This is a schematic diagram illustrating the extraction of hierarchical features of Chinese characters, vocabulary, and grammar. Detailed Implementation

[0064] The present invention will be further described below with reference to the accompanying drawings and embodiments. The embodiments of the present invention include, but are not limited to, the following embodiments.

[0065] See Figure 1 The left side of the diagram illustrates the model training process of this invention, which mainly includes: dataset partitioning, language level feature extraction, loading BERT pre-training parameters, deep fusion of language level features, and setting the loss function and training parameters required for model training. The right side illustrates the prediction process of this invention, which mainly includes: loading model parameters, inputting the text to be classified, extracting language level features, deep fusion of language level features, and outputting the predicted level.

[0066] See Figure 2 For details of the main framework and process of this invention, please refer to [link / reference]. Figure 2 (a) For details on embedding language level features into the embedding layer, please refer to [link / reference]. Figure 2 (b) For details on integrating language level features into the self-attention layer, please refer to [link / reference]. Figure 2 (c) In the framework diagram, the input article is fed into the language level feature extraction module to extract language level features of Chinese characters, vocabulary, and grammar; then it enters the feature fusion module, which integrates the language level features in the embedding layer and self-attention layer of the BERT model. Specifically, the level features of "Chinese characters" and "vocabulary" are fused in the embedding layer, and the level features of "grammar" are fused in the first 7 layers of the self-attention layer; the difficulty level probability distribution of the article is obtained through the Softmax layer, and finally the prediction result of the article difficulty level is obtained.

[0067] See Figure 3 This invention extracts character, vocabulary, and grammar level features and designs diverse storage methods to comprehensively characterize the readability of text in terms of characters, vocabulary, and grammar. In vocabulary level feature extraction, this invention references the n-gram concept, extracting words from the article with a maximum word length of N, and gradually shortening the word length until the word is found in the outline. This operation is performed on the article from left to right. Considering that the maximum word length in the vocabulary difficulty level outline of the *International Chinese Language Education Chinese Proficiency Level Standards* is 4, this invention sets N to 4. In grammar level feature extraction, the grammar difficulty level outline provided by the *International Chinese Language Education Chinese Proficiency Level Standards* contains 572 grammar points. By constructing regular expressions, this invention completed the matching of 474 of these grammar points, achieving a matching completion rate of 82.87%. If a position involves multiple grammar points, the highest grammar difficulty level is taken as the grammar level for that position.

[0068] Specifically, the training steps of the present invention are as follows:

[0069] Step 1: Dataset Partitioning

[0070] This invention uses 5721 articles from a Chinese text readability grading dataset, divided in an 8:1:1 ratio into a training set of 4576 articles, a validation set of 572 articles, and a test set of 573 articles. Each article has a correct difficulty level label. The training set texts are used to train the model; the validation set texts are used to adjust the model's hyperparameters to prevent overfitting to the training set; and the test set texts are used to evaluate the model's generalization performance.

[0071] Step 2: Language Level Feature Extraction

[0072] The language-level feature extraction method described above is applied to all texts in the dataset to extract the character, vocabulary, and grammar-level feature vectors and matrices. Since the character and vocabulary-level features are fused at the embedding layer, and the grammar-level features are fused at the self-attention layer, this part only needs to obtain the character and vocabulary-level feature vectors and the grammar-level feature matrix.

[0073] Step 3: Use the BERT model for word segmentation

[0074] The articles in the dataset are segmented using the BERT model's BertTokenizer, mapping all the text in the articles to the BERT model's vocabulary for input into subsequent model layers.

[0075] Step 4: Load BERT pre-trained parameters

[0076] The bert-base-chinese pre-trained parameters from the Hugging Face website are used as the initialization parameters for this invention.

[0077] Step 5: Embedding layer incorporates language level features

[0078] The character and vocabulary level feature vectors extracted in step 2 are converted into level feature embedding vector representations through an embedding layer, i.e., t m =Embedding(V m ), m∈{c,w,g}. Where V m For the hierarchical feature vector, t m The hierarchical feature embedding is then performed. Subsequently, the language hierarchical feature embedding (including Chinese character hierarchical embedding, lexical hierarchical embedding, and grammatical hierarchical embedding) is added to the original three embedding parts of BERT to obtain the article's embedding representation, i.e. t j =t token +t seg +t pos +t char +t word +t gram Among them, t j It is the embedding vector representation of the j-th character in the article.

[0079] Step 6: Integrating language hierarchy features into the self-attention layer

[0080] Based on the attention score calculated by the BERT model, the language level feature matrix is ​​incorporated into its attention calculation, i.e., A = Q * K. T +M c +M w +M g Among them, M c M is the feature matrix of Chinese characters; w M is the vocabulary feature matrix; g This is the grammatical feature matrix.

[0081] Step 7: Construct the loss function. Choose the cross-entropy loss function. The article set T contains N samples, and the number of classes is |C|. The cross-entropy loss function can be expressed as: Where θ is the model parameter, y i,j This represents the true probability that the i-th sample belongs to the j-th class. This represents the probability that the model predicts the sample belongs to the j-th class. The training objective is to minimize the value of the cross-entropy loss function, i.e., θ. * =argmin θ J(θ).

[0082] Step 8: Training Setup. Use the AdamW optimizer during training, set the batch size to 32, and the learning rate to 5e-5. Train for 20 iterations, and select the model that achieves the highest accuracy on the validation set as the optimal model.

[0083] Specifically, the prediction steps of the present invention are as follows:

[0084] Step 1: Model Preparation. Load the trained model parameters and structure to quickly process the input text; load the required level outline data for language level feature extraction.

[0085] Step 2: Language Level Feature Extraction. The language level feature extraction method described above is used to extract the Chinese character level feature vector, vocabulary level feature vector, and grammar level feature matrix from the input text.

[0086] Step 3: Tokenize using the BERT model. The input text is segmented using the BERT model's BertTokenizer and mapped to the BERT model's vocabulary for input into subsequent model layers.

[0087] Step 4: Obtain the output layer results. Input the word segmentation results and language level features of the input text into the trained model. The model will automatically incorporate language level features into the embedding layer and the self-attention layer, and obtain the output layer results.

[0088] Step 5: Predict the difficulty level. Using the Softmax function, the output layer result from Step 4 is mapped to the difficulty level space, and the level with the highest probability is taken as the predicted level of the input text.

[0089] Example 1

[0090] The following is an example of the readability classification of input text in this invention. Assuming the input text is "Frequently saying 'thank you' to others not only makes one appear polite and well-mannered, but also promotes physical and mental health. Scientific research shows that people who are often grateful not only have a higher sense of happiness and better health, but also get along better with others. Furthermore, having a grateful heart helps people maintain a positive and optimistic attitude towards life, allowing them to remain calm and composed when facing pressure and difficulties, and to be full of confidence in the future."

[0091] (1) Model preparation

[0092] Load the trained model parameters and model structure, along with the required level outline data, for language level feature extraction.

[0093] (2) Language level feature extraction

[0094] Using the Chinese character, vocabulary, and grammar proficiency outlines of the "Standard", extract the Chinese character proficiency feature vector of the input text as V c =(2, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 1, 3, 1, 2, 1, 3, 1, 1, 2, 5, 0, …, 0), the vocabulary proficiency feature vector V w =(2, 2, 1, 1, 1, 1, 0, 1, 1, 0, 0, 3, 3, 1, 3, 1, 3, 3, 1, 5, 5, 0, …, 0), and the grammar proficiency feature matrix

[0095] (3) BERT tokenization

[0096] Use BertTokenizer to tokenize the input text and map it to the BERT vocabulary. For the above input text sample, the tokenization result is: ['Often', 'tell', 'others', '[UNK]', 'thank', '[UNK]', ',', 'not', 'only', 'will', 'make', 'people', 'appear', 'polite', 'and', 'cultured', ',', 'but', 'also', 'can', …].

[0097] (4) Input the trained model

[0098] Input the extracted language proficiency features and the BERT tokenization result into the trained model. The model will automatically complete the embedding representation of the input text in the embedding layer and add the Chinese character and vocabulary grammar feature vectors to this embedding representation; and complete the integration of the grammar feature matrix in the self-attention layer.

[0099] (5) Obtain the result of the output layer

[0100] For the above input text sample, the Softmax result of the model output layer is tensor([[-1.8978, -2.1510, -2.2412, -1.0841, 8.0487, 0.3150]]), and take the difficulty level with the highest probability as the predicted difficulty level. Therefore, in this example, the predicted difficulty level of the model is level 5.

[0101] The above embodiments are only illustrative of the principles and effects of the present invention and are not used to limit the present invention. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed by the present invention should still be covered by the claims of the present invention.

Claims

1. A method for classifying the readability of Chinese text based on multi-level language feature fusion, characterized in that, A Chinese text readability classification model based on multi-level language feature fusion is constructed using a language level feature extraction module and a multi-level feature fusion module. The trained Chinese text readability classification model is used to predict the difficulty level of input text. The Chinese text readability classification model includes model training and model prediction. Model training uses Chinese text readability classification data to train model parameters. The training process specifically includes: dataset partitioning, language level feature extraction, word segmentation using the BERT model, loading BERT pre-trained parameters, incorporating language level features into the embedding layer, incorporating language level features into the self-attention layer, constructing the loss function, and setting training parameters. Model prediction uses the trained Chinese text readability classification model to predict the difficulty level of input text. The prediction process specifically includes: model preparation, language level feature extraction, word segmentation using the BERT model, obtaining the output layer results, and obtaining the predicted difficulty level. The language level feature extraction module uses the level outline of Chinese characters, vocabulary, and grammar to locate Chinese characters, vocabulary, and grammar points in the input text, extract the level information of the current position, convert the text into three level features of Chinese characters, vocabulary, and grammar, and store them in the form of a combination of feature vectors and feature matrices. The multi-level feature fusion module deeply fuses the language-level features of Chinese characters, vocabulary, and grammar in the article at both the embedding layer and the self-attention layer, specifically including the following steps: 3-1: Embedding layer integrates language hierarchy features A language-level feature embedding component, including Chinese character-level embedding, vocabulary-level embedding, and grammar-level embedding, is added to the BERT model's embedding layer. The language-level feature embedding component is then added to the original three embedding components of BERT to obtain the article's embedding representation as shown in equation (a) below. : （a）； in, For the first in the article The embedding vector representation of each character. ;in, These are word embedding vectors, sentence segmentation embedding vectors, position embedding vectors, Chinese character-level feature embedding vectors, vocabulary-level feature embedding vectors, and grammar-level feature embedding vectors, respectively. The language level feature embedding portion is calculated by the following formula (b): ， （b）； in, For hierarchical feature vectors; Embedding for hierarchical features; These respectively represent Chinese characters, vocabulary, and grammatical features; 3-2: Integrating Language Hierarchy Features into the Self-Attention Layer Language level feature information is incorporated into the self-attention layer of the Transformer layer in the BERT model, and the language level feature matrix is ​​incorporated into the attention calculated by the following equation (c) when the BERT model calculates the attention score. : （c）； in, This is a matrix of Chinese character features; For the vocabulary feature matrix; This is the grammatical feature matrix; For query matrix; The key matrix; The Chinese character level feature matrix Then the Chinese character level feature vector The values ​​on the diagonal of the matrix are the values ​​on the diagonal of the matrix. This matrix is ​​a diagonal matrix with a size of . .

2. The Chinese text readability classification method based on multi-level language feature fusion according to claim 1, characterized in that, The language level feature extraction module extracts three level features: Chinese characters, vocabulary, and grammar. Specifically, it includes the following steps: 2-1: Extraction of hierarchical features of Chinese characters The text is traversed, and the input article is converted into an integer vector of the same length as the input article. If a non-Chinese character is encountered, the vector value at that position is directly assigned "0". If a Chinese character is encountered, its corresponding difficulty level is found according to the Chinese character difficulty level outline, and the vector value at that position is assigned the level number. If the Chinese character cannot be found in the Chinese character difficulty level outline, the vector value at that position is assigned "0". The integer value of each dimension in the vector ranges from 0 to 7, thus obtaining the Chinese character level feature vector of the article. ,in For the first The level of each Chinese character The length of the article; 2-2: Lexical Level Feature Extraction After segmenting the article into words, each word is matched against the difficulty level in the lexicon. The vector values ​​of unmatched characters in the article are assigned "0", while the vector values ​​of words matched in the lexicon are assigned the difficulty level number. The input article is then converted into an integer vector of the same length as the original text. Each dimension of the vector takes integer values ​​ranging from 0 to 7, thus constructing a vocabulary-level feature vector. ; 2-3: Syntax Level Feature Extraction Regular expressions are used to match grammar points in the article. Based on the grammar difficulty level outline, the corresponding difficulty level of the matched grammar points is found. The vector value of the non-contiguous grammar marker of the grammar point in the article is assigned as the level number, and the grammar features are represented by a class adjacency matrix.

3. The Chinese text readability classification method based on multi-level language feature fusion according to claim 1, characterized in that, The model training specifically includes the following steps: 4-1: Dataset Partitioning The Chinese text readability rating data was divided into a training set, a validation set, and a test set. The training set text was used... The validation set text is used to adjust the model's hyperparameters to prevent the model from overfitting to the training set; the test set text is used to evaluate the model's generalization performance. 4-2: Language Level Feature Extraction The language level feature extraction method of the language level feature extraction module is used to extract Chinese character, vocabulary level feature vector and grammar level feature matrix from all texts in the dataset. 4-3: Using the BERT model for word segmentation The articles in the dataset are segmented using the BERT model's BertTokenizer, and all the text in the articles is mapped to the BERT model's vocabulary. 4-4: Loading BERT pre-trained parameters Load the BERT pre-trained parameters as the initialization parameters of the model; 4-5: Embedding layer incorporates language hierarchy features A method of incorporating language level features into the embedding layer of a multi-level feature fusion module is adopted. The extracted Chinese character and vocabulary level feature vectors are converted into level feature embedding vector representations through the embedding layer. The language level feature embedding part is added to the original three embedding parts of BERT to obtain the embedded representation of the article. 4-6: Integrating language hierarchy features into the self-attention layer A method that integrates language level features into the attention layer using a multi-level feature fusion module is adopted. Based on the attention score calculated by the BERT model, the language level feature matrix is ​​incorporated into its attention. Calculate and obtain an attention score with language level feature information, the attention... The calculation is the same as that described in equation (c) in step 3-2; 4-7: Constructing the loss function Use the cross-entropy loss function of equation (d) below as the loss function for model training: （d）； in, For a collection of articles The number of samples included; Number of categories; These are model parameters; For the first The sample belongs to the first The true probability of each category; The model predicts that the sample belongs to the first... The probability of each category; the training objective is set to minimize the value of the cross-entropy loss function, which is expressed by the following equation (g): （g）； 4-8: Training Settings The AdamW optimizer was used for model training, with a batch size of 32 and a learning rate of 5e to 5. The model was trained for 20 iterations, and the model that achieved the highest accuracy on the validation set was selected as the optimal model.

4. The Chinese text readability classification method based on multi-level language feature fusion according to claim 1, characterized in that, The model prediction specifically includes the following steps: 5-1: Model Preparation Load the Chinese character, vocabulary, and grammar level outlines into memory, and load the trained model results and their parameters into memory; 5-2: Language Level Feature Extraction The language level feature extraction module is used to extract three level features of Chinese characters, vocabulary and grammar. The input text is then used to extract Chinese character level feature vectors, vocabulary level feature vectors and grammar level feature matrices. 5-3: Using the BERT model for word segmentation The input text is segmented using the BERT model's BertTokenizer and mapped to the BERT model's vocabulary. 5-4: Obtain the output layer results The word segmentation results and language level features of the input text are input into the trained model. The model will automatically incorporate language level features into the embedding layer and the self-attention layer to obtain the output layer results. 5-5: Prediction Difficulty Level Using the Softmax function, the output layer results are mapped to a difficulty level space, and the level with the highest probability is used as the predicted level of the input text.

Images