Chinese text difficulty classification method and system and storage medium

Abstract

The invention discloses a Chinese text difficulty classification method and system and a storage medium. The method comprises the following steps: performing text word segmentation processing on an obtained Chinese text to be processed; obtaining the Embedding expression of each Chinese vocabulary in the Chinese vocabulary set, and calculating linguistic indexes; the Embedding vectors and linguistic indexes are combined, and a feature matrix is constructed and obtained; processing the feature matrix by using a convolutional neural network, and extracting local features; processing the feature matrix by using a pre-trained Transform model, and extracting global features; and fusing the local features and the global features to form final feature representation of the text, and inputting the final feature representation into a classifier for text difficulty classification to obtain a Chinese text difficulty classification result. The embodiment of the invention can be suitable for various scenes needing to evaluate the text readability, is high in classification efficiency and high in classification accuracy, and can be widely applied to the technical field of computers.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, system, and storage medium for classifying the difficulty of Chinese text. Background Technology

[0002] With the development of network technology, massive amounts of unstructured data such as text, images, and videos are generated on the internet every day. Text data can be processed using modern natural language processing (NLP) techniques to extract more valuable information hidden within it. For a long time, the mainstream NLP techniques were rule-based, ranging from various syntactic analyses to semantic analyses. It was only with the development of the internet and the generation of massive amounts of corpus data that statistical natural language processing gradually emerged.

[0003] Traditional readability formulas can perform well in certain specific use cases. However, when generalized to more general scenarios, they suffer from problems such as insufficient feature selection and unsatisfactory results from the linear models used. Traditional methods rely on manual grading, which is a massive and time-consuming task given the vast amount of Chinese text with diverse topics and styles. Moreover, manual grading is often highly subjective, making it difficult to reach a consensus during review.

[0004] Existing methods generally suffer from the problem of relying solely on local or global features. For example, traditional methods based solely on word frequency and sentence length tend to fail when faced with synonym substitution and complex long sentence structures; while models based solely on Transformer perform poorly when processing short texts or relying on local details. Summary of the Invention

[0005] The main objective of this invention is to propose a Chinese text difficulty classification method, system, and storage medium. By combining CNN and Transformer, it considers both local and global features, making it applicable to various scenarios that require evaluating text readability. It also boasts high classification efficiency and accuracy.

[0006] To achieve the above objectives, one aspect of this invention proposes a method for classifying the difficulty of Chinese text, comprising the following steps:

[0007] The acquired Chinese text to be processed is subjected to text segmentation to obtain a set of Chinese words;

[0008] Obtain the embedding representation of each Chinese word in the Chinese vocabulary set to obtain an embedding vector set;

[0009] Based on the Embedding vector set, linguistic indicators are calculated; these linguistic indicators are used to characterize the lexical, sentence, discourse, and syntactic complexity features of the text.

[0010] The embedding vector of each Chinese word is combined with the linguistic indicators to construct a feature matrix;

[0011] A convolutional neural network is used to process the feature matrix and extract local features; these local features are used to characterize local patterns in the data and the relationships between adjacent elements.

[0012] The feature matrix is ​​processed using a pre-trained Transformer model to extract global features; these global features are used to characterize long-distance dependencies and global relationships in the word sequence.

[0013] Local and global features are fused to form the final feature representation of the text; the final feature representation is then input into a classifier to classify the text difficulty, resulting in the Chinese text difficulty classification result.

[0014] In some embodiments, the step of performing text segmentation on the acquired Chinese text to obtain a Chinese vocabulary set includes the following steps:

[0015] Call the Kimi large model to select target word segmentation tool;

[0016] Based on the word segmentation tool, the acquired Chinese text to be processed is segmented into words using a word segmentation strategy;

[0017] After word segmentation is completed, stop words are filtered out.

[0018] The words obtained from word segmentation are tagged with their parts of speech to obtain a set of Chinese vocabulary.

[0019] In some embodiments, obtaining the embedding representation of each Chinese word in the Chinese vocabulary set to obtain an embedding vector set includes the following steps:

[0020] The pre-trained embedding model provides an embedding representation for each word with a dimension of 3072.

[0021] For each word, call the API or interface of the Embedding model, input the word and get the corresponding Embedding vector.

[0022] In some embodiments, calculating the linguistic index based on the Embedding vector set includes the following steps:

[0023] The average word frequency is determined by calculating the average frequency of occurrence of words and the ratio of high-frequency words to low-frequency words based on a word frequency dictionary or corpus; the lexical diversity is determined by calculating the type-tag ratio by dividing the number of different words by the total number of words; and then the lexical complexity index is determined based on the average word frequency and the lexical diversity.

[0024] Sentence complexity metrics are determined by calculating average sentence length, number of subordinate clauses, and their proportion.

[0025] The text structure indicators are determined by calculating the total number of words, the total number of sentences, and the keyword density.

[0026] Syntactic complexity metrics are determined by calculating dependency tree depth and the number of syntactic clauses.

[0027] The various indicators are combined into a feature vector;

[0028] The linguistic indicators include at least average word frequency, type-to-tag ratio, average sentence length, clause ratio, topic word density, dependency tree depth, and number of syntactic clauses.

[0029] In some embodiments, combining the embedding vector of each Chinese word with the linguistic indicators to construct a feature matrix includes the following steps:

[0030] For each word's embedding vector, 23 linguistic indicators are appended to form an extended vector;

[0031] The feature matrix is ​​constructed by using each extended vector as the i-th column of the feature matrix.

[0032] In some embodiments, the process of using a convolutional neural network to process the feature matrix and extract local features includes the following steps:

[0033] By taking the feature matrix as input and adding a channel dimension, the corresponding input shape is determined.

[0034] Temporal convolution of the convolutional layer performs one-dimensional convolution in the temporal dimension to capture local patterns in the word sequence, and channel convolution performs one-dimensional convolution in the channel dimension to capture the correlation between features.

[0035] Constructing activation functions increases the model's non-linear representation capability, and adding batch normalization layers after convolutional layers stabilizes the training process and accelerates convergence;

[0036] Constructing pooling layers involves using average pooling or max pooling layers to downsample features in both time and channel dimensions, and setting the pooling kernel size and stride as needed.

[0037] After convolution and pooling layers, a feature tensor is obtained, and finally the feature is output; the feature tensor contains local patterns and correlation information between adjacent words;

[0038] The expression for extracting local features is: F CNN =Pooling(Activation(Conv(E))), where Conv represents the convolution operation, Activation represents the activation function, and Pooling represents the pooling operation;

[0039] The convolutional neural network has a kernel size of 6 in the time dimension to capture local features of three adjacent words; the activation function is ReLU, and the pooling layer is a max pooling layer with a kernel size of 2.

[0040] In some embodiments, the process of using a pre-trained Transformer model to process the feature matrix and extract global features includes the following steps:

[0041] Input data preparation specifically involves using features extracted by CNN as input to the Transformer and adding positional encoding to the input to determine the position of the words recognized by the model in the sequence.

[0042] The Transformer architecture is constructed by building a multi-head self-attention mechanism, a feedforward neural network, residual connections, and layer normalization.

[0043] Stacking Transformer layers involves stacking multiple Transformer layers according to task requirements to enhance the model's expressive power.

[0044] After processing by the Transformer, global features are obtained; these global features contain the dependencies between distant words in the sequence.

[0045] The Transformer uses a two-layer stacked Transformer encoder, with each layer including an eight-head self-attention mechanism and a feedforward neural network, and a hidden dimension of 512.

[0046] Another aspect of this invention provides a Chinese text difficulty classification system, including:

[0047] The first module is used to perform text segmentation on the acquired Chinese text to be processed, and obtain a set of Chinese words.

[0048] The second module is used to obtain the embedding representation of each Chinese word in the Chinese word set, and to obtain the embedding vector set;

[0049] The third module is used to calculate linguistic indicators based on the Embedding vector set; the linguistic indicators are used to characterize the lexical, sentence, discourse, and syntactic complexity features of the text.

[0050] The fourth module is used to combine the embedding vector of each Chinese word with the linguistic indicators to construct a feature matrix;

[0051] The fifth module is used to process the feature matrix using a convolutional neural network to extract local features; these local features are used to characterize local patterns in the data and the relationships between adjacent elements.

[0052] The sixth module is used to process the feature matrix using a pre-trained Transformer model to extract global features; these global features are used to represent long-distance dependencies and global relationships in the word sequence.

[0053] The seventh module is used to fuse local and global features to form the final feature representation of the text; the final feature representation is then input into a classifier to classify the text difficulty, resulting in a Chinese text difficulty classification result.

[0054] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0055] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0056] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0057] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a method, system, and storage medium for classifying the difficulty of Chinese text. This scheme involves performing text segmentation on the acquired Chinese text to be processed to obtain a set of Chinese words; obtaining the embedding representation of each Chinese word in the set of Chinese words to obtain a set of embedding vectors; calculating linguistic indicators based on the set of embedding vectors; the linguistic indicators are used to characterize the lexical, sentence, discourse, and syntactic complexity features of the text; combining the embedding vectors of each Chinese word with the linguistic indicators to construct a feature matrix; processing the feature matrix using a convolutional neural network to extract local features; the local features are used to characterize local patterns and relationships between adjacent elements in the data; processing the feature matrix using a pre-trained Transformer model to extract global features; the global features are used to characterize long-distance dependencies and global relationships in the word sequence; fusing the local and global features to form the final feature representation of the text; and inputting the final feature representation into a classifier to classify the text difficulty to obtain the Chinese text difficulty classification result. The embodiments of the present invention are applicable to various scenarios that require evaluation of text readability, and have high classification efficiency and high classification accuracy. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0059] Figure 2 This is a flowchart of the overall steps provided in the embodiments of the present invention;

[0060] Figure 3 This is a flowchart of the Chinese word segmentation process provided in an embodiment of the present invention;

[0061] Figure 4 This is a flowchart of the word embedding process provided in the embodiments of the present invention;

[0062] Figure 5 This is a flowchart of the language complexity analysis process provided in an embodiment of the present invention;

[0063] Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0065] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0066] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0068] The Chinese text difficulty classification method, system, and storage medium provided in this invention relate to the field of computer technology. The Chinese text difficulty classification method provided in this invention can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the Chinese text difficulty classification method, but is not limited to the above forms.

[0069] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0070] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0071] Server 101 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0072] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0073] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0074] Exemplary based on Figure 1 The implementation environment shown in this embodiment of the invention provides a Chinese text difficulty classification method. The following description uses the application of this Chinese text difficulty classification method in server 101 as an example. It can be understood that this method can also be applied in terminal 102.

[0075] Reference Figure 2 , Figure 2 This is a flowchart illustrating a Chinese text difficulty classification method applied to a server, provided as an embodiment of the present invention. The executing entity of this method can be any of the aforementioned computer devices (including servers or terminals). (Refer to...) Figure 2 The method may include the following steps:

[0076] The acquired Chinese text to be processed is subjected to text segmentation to obtain a set of Chinese words;

[0077] Obtain the embedding representation of each Chinese word in the Chinese vocabulary set to obtain an embedding vector set;

[0078] Based on the Embedding vector set, linguistic indicators are calculated; these linguistic indicators are used to characterize the lexical, sentence, discourse, and syntactic complexity features of the text.

[0079] The embedding vector of each Chinese word is combined with the linguistic indicators to construct a feature matrix;

[0080] A convolutional neural network is used to process the feature matrix and extract local features; these local features are used to characterize local patterns in the data and the relationships between adjacent elements.

[0081] The feature matrix is ​​processed using a pre-trained Transformer model to extract global features; these global features are used to characterize long-distance dependencies and global relationships in the word sequence.

[0082] Local and global features are fused to form the final feature representation of the text; the final feature representation is then input into a classifier to classify the text difficulty, resulting in the Chinese text difficulty classification result.

[0083] In some embodiments, the step of performing text segmentation on the acquired Chinese text to obtain a Chinese vocabulary set includes the following steps:

[0084] Call the Kimi large model to select target word segmentation tool;

[0085] Based on the word segmentation tool, the acquired Chinese text to be processed is segmented into words using a word segmentation strategy;

[0086] After word segmentation is completed, stop words are filtered out.

[0087] The words obtained from word segmentation are tagged with their parts of speech to obtain a set of Chinese vocabulary.

[0088] In some embodiments, obtaining the embedding representation of each Chinese word in the Chinese vocabulary set to obtain an embedding vector set includes the following steps:

[0089] The pre-trained embedding model provides an embedding representation for each word with a dimension of 3072.

[0090] For each word, call the API or interface of the Embedding model, input the word and get the corresponding Embedding vector.

[0091] In some embodiments, calculating the linguistic index based on the Embedding vector set includes the following steps:

[0092] The average word frequency is determined by calculating the average frequency of occurrence of words and the ratio of high-frequency words to low-frequency words based on a word frequency dictionary or corpus; the lexical diversity is determined by calculating the type-tag ratio by dividing the number of different words by the total number of words; and then the lexical complexity index is determined based on the average word frequency and the lexical diversity.

[0093] Sentence complexity metrics are determined by calculating average sentence length, number of subordinate clauses, and their proportion.

[0094] The text structure indicators are determined by calculating the total number of words, the total number of sentences, and the keyword density.

[0095] Syntactic complexity metrics are determined by calculating dependency tree depth and the number of syntactic clauses.

[0096] The various indicators are combined into a feature vector;

[0097] The linguistic indicators include at least average word frequency, type-to-tag ratio, average sentence length, clause ratio, topic word density, dependency tree depth, and number of syntactic clauses.

[0098] In some embodiments, combining the embedding vector of each Chinese word with the linguistic indicators to construct a feature matrix includes the following steps:

[0099] For each word's embedding vector, 23 linguistic indicators are appended to form an extended vector;

[0100] The feature matrix is ​​constructed by using each extended vector as the i-th column of the feature matrix.

[0101] In some embodiments, the process of using a convolutional neural network to process the feature matrix and extract local features includes the following steps:

[0102] By taking the feature matrix as input and adding a channel dimension, the corresponding input shape is determined.

[0103] Temporal convolution of the convolutional layer performs one-dimensional convolution in the temporal dimension to capture local patterns in the word sequence, and channel convolution performs one-dimensional convolution in the channel dimension to capture the correlation between features.

[0104] Constructing activation functions increases the model's non-linear representation capability, and adding batch normalization layers after convolutional layers stabilizes the training process and accelerates convergence;

[0105] Constructing pooling layers involves using average pooling or max pooling layers to downsample features in both time and channel dimensions, and setting the pooling kernel size and stride as needed.

[0106] After convolution and pooling layers, a feature tensor is obtained, and finally the feature is output; the feature tensor contains local patterns and correlation information between adjacent words;

[0107] The expression for extracting local features is: F CNN =Pooling(Activation(Conv(E))), where Conv represents the convolution operation, Activation represents the activation function, and Pooling represents the pooling operation;

[0108] The convolutional neural network has a kernel size of 6 in the time dimension to capture local features of three adjacent words; the activation function is ReLU, and the pooling layer is a max pooling layer with a kernel size of 2.

[0109] In some embodiments, the process of using a pre-trained Transformer model to process the feature matrix and extract global features includes the following steps:

[0110] Input data preparation specifically involves using features extracted by CNN as input to the Transformer and adding positional encoding to the input to determine the position of the words recognized by the model in the sequence.

[0111] The Transformer architecture is constructed by building a multi-head self-attention mechanism, a feedforward neural network, residual connections, and layer normalization.

[0112] Stacking Transformer layers involves stacking multiple Transformer layers according to task requirements to enhance the model's expressive power.

[0113] After processing by the Transformer, global features are obtained; these global features contain the dependencies between distant words in the sequence.

[0114] The Transformer uses a two-layer stacked Transformer encoder, with each layer including an eight-head self-attention mechanism and a feedforward neural network, and a hidden dimension of 512.

[0115] The specific implementation process of this invention will be described in detail below using a specific application scenario as an example:

[0116] The Chinese text difficulty classification method provided in this embodiment of the invention includes the following steps:

[0117] Step 1: Text segmentation processing.

[0118] refer to Figure 3 In this embodiment of the invention, a segment of input Chinese text is processed by word segmentation. The purpose of word segmentation is to divide a continuous text sequence into independent words in order to facilitate subsequent feature extraction and analysis.

[0119] 1. Selection of word segmentation tools: Use a combination of Chinese word segmentation tools (jieba, THULAC, and HanLP). These tools can accurately segment the text based on the characteristics of the Chinese language. If the segmentation methods of multiple tools are inconsistent, the Kimi large model will be used to select the optimal segmentation method.

[0120] 2. Word Segmentation Strategy: Use the exact mode or the full mode for word segmentation. The exact mode can obtain accurate segmentation results and avoid excessive redundant words.

[0121] 3. Stop Word Filtering: After word segmentation, it is possible to consider filtering common stop words such as "de", "le", "zai", etc. to reduce interference to the model.

[0122] 4. Part-of-Speech Tagging: Optionally, perform part-of-speech tagging on each word to provide additional information for subsequent calculation of linguistic indicators.

[0123] 5. Processing Example: Suppose the input text is: "Machine learning is a branch of artificial intelligence." After word segmentation, the vocabulary list obtained is: ["Machine learning", "is", "artificial intelligence", "de", "a", "branch"].

[0124] After the first step, assume that the embodiments of the present invention obtain n Chinese words, and then enter the processing of the second step.

[0125] Step 2: Obtain the Embedding representation of the words.

[0126] Refer to Figure 4 , for each Chinese word obtained by word segmentation, obtain its Embedding representation on OpenAI. Embedding is a technique that maps words to a high-dimensional vector space for capturing the semantic and syntactic information of words.

[0127] 1. Selection of Embedding Model: Use the pre-trained Embedding model provided by OpenAI, such as GPT-3 or other models suitable for Chinese. These models can provide the Embedding representation of each word, with a dimension of 3072.

[0128] 2. Obtain Embedding: For each word, call the API or interface of the model, input the word, and obtain the corresponding Embedding vector e i ∈R 3072 .

[0129] 3. Notes:

[0130] 1). Out-of-Vocabulary (OOV) Problem: For words not included in the Embedding model, processing measures need to be taken, such as using the UNK token or trying other models.

[0131] 2). Batch Processing: To improve efficiency, the Embeddings of words can be obtained in batches to reduce the number of API calls.

[0132] 4. Example: For a list of words ["machine learning", "is", "artificial intelligence", "of", "a", "branch"], obtain the corresponding embedding vectors e1, e2, ..., e n .

[0133] Through this step, this embodiment of the invention obtains corresponding embedding vectors for n words, with each vector having a dimension of 3072.

[0134] Step 3: Calculate linguistic indicators.

[0135] refer to Figure 5 Based on the acquisition of lexical embeddings, 23 linguistic metrics are calculated to capture the lexical, sentence, discourse, and syntactic complexity features of the text.

[0136] 1. Lexical complexity index:

[0137] 1) Average word frequency: Calculates the average frequency of occurrence of words, the ratio of high-frequency words to low-frequency words. This can be done using word frequency dictionaries or corpora.

[0138] 2) Lexical diversity (TTR): Type-Token Ratio, which is the number of different words divided by the total number of words.

[0139] 2. Sentence complexity index:

[0140] 1) Average sentence length: The average length of a sentence can be either the average number of characters or the average number of words.

[0141] 2) Number and percentage of clauses: Count the number of clauses in a sentence and their percentage of the total number of sentences.

[0142] 3. Text structure indicators:

[0143] 1) Total word count: The total number of words in the text.

[0144] 2) Total number of sentences: The total number of sentences in the text.

[0145] 3) Topic density: The density of keywords, which can be calculated using TF-IDF or keyword extraction algorithms.

[0146] 4. Syntactic complexity index:

[0147] 1) Dependency tree depth: Using dependency parsing, the dependency tree depth of a sentence is calculated, reflecting the complexity of the syntactic structure.

[0148] 2) Number of syntactic clauses: Count the number of clauses in the syntactic structure.

[0149] 5. Calculation method of the indicator:

[0150] 1) Corpus and tools: Use linguistic toolkits, such as NLTK, Stanford Parser, or self-developed tools, to perform syntactic analysis and index calculation.

[0151] 2) Data standardization: Standardize the calculated indicators, such as by normalization or Z-score standardization, to eliminate the influence of dimensions.

[0152] 6. Formation of the indicator vector: The 23 linguistic indicators are combined into a feature vector l∈R. 23 .

[0153] Step 4: Construct the feature matrix.

[0154] The word embeddings and linguistic indicators are combined to form a feature matrix of size 3095×n.

[0155] 1. Combining embedding and linguistic indicators:

[0156] 1) Expanding the Embedding Vector: For each word, the Embedding vector e i ∈R 3072 Then, 23 linguistic indicators l are added to form an extended vector e′. i ∈R 3095 .

[0157] 2) Since linguistic metrics are applied to the entire text and are global, the same linguistic metric vector l is attached to each word.

[0158] 2. Construction of the feature matrix:

[0159] 1) Column composition: Each extended vector e′ i It is the i-th column of the feature matrix.

[0160] 2) Matrix representation: Constructing the characteristic matrix E∈R 3095×n , where E=[e′1,e′2,...,e′ n ].

[0161] 3. The correspondence between the matrix and the original data:

[0162] 1) Number of channels: 3095 corresponds to the number of channels in different dimensions, similar to different electrode signals in EEG data.

[0163] 2) Number of data points: n corresponds to the number of data points (samples) in EEG, which can represent the sampling of the signal in the time dimension.

[0164] Step 5: Feature Extraction – CNN Extraction of Local Features.

[0165] Convolutional Neural Networks (CNNs) are used to process feature matrices and extract local features. CNNs can capture local patterns and relationships between adjacent elements in data.

[0166] 1. Input data preparation:

[0167] 1) Input shape: The feature matrix Treat it as input, add a channel dimension, and the shape is

[0168] 2. Convolutional layer design:

[0169] 1) Temporal Convolution:

[0170] 11) Kernel size: Set the kernel size to (1, k) t ), where k t It is the kernel size in the time dimension.

[0171] 12) Convolution operation: Perform one-dimensional convolution in the time dimension (n) to capture local patterns in the word sequence.

[0172] 2) Channel Convolution:

[0173] 21) Kernel Size: Set the kernel size to (k c ,1), where k c It is the kernel size in the channel dimension.

[0174] 22) Convolution operation: Perform one-dimensional convolution on the channel dimension (3095) to capture the correlation between features.

[0175] 3. Activation functions and regularization:

[0176] 1) Activation function: Use non-linear activation functions such as ReLU or ELU to increase the non-linear representation capability of the model.

[0177] 2) Batch Normalization: Add a batch normalization layer after the convolutional layer to stabilize the training process and accelerate convergence.

[0178] 4. Pooling layer:

[0179] 1) Pooling operation: Use average pooling or max pooling layers to downsample features in the time and channel dimensions, reduce the number of parameters, and prevent overfitting.

[0180] 2) Pooling kernel size: Set the pooling kernel size and step size as needed.

[0181] 5. Output characteristics:

[0182] 1) Feature Tensor: After convolution and pooling layers, the feature tensor F is obtained. CNN It contains local patterns and information about the relationships between adjacent words.

[0183] 6. Illustrated formula:

[0184] F CNN =Pooling(Activation(Conv(E)))

[0185] Here, Conv represents convolution operation, Activation represents activation function, and Pooling represents pooling operation.

[0186] Step 6: Feature Extraction – Transformer Extraction of Global Features.

[0187] By using a pre-trained Transformer model, global features are extracted to capture long-distance dependencies and global relationships in word sequences.

[0188] 1. Input data preparation:

[0189] 1) Feature Tensor: The feature F extracted using CNN CNN As input to the Transformer.

[0190] 2) Position encoding: Add position encoding to the input to help the model identify the position of words in the sequence.

[0191] 2. Transformer architecture:

[0192] 1) Multi-head self-attention mechanism:

[0193] 11) Calculate the query (Q), key (K), and value (V):

[0194] Q = F CNN W Q K = F CNN W K V = F CNN W V

[0195] Among them, W Q W K W V is a learnable parameter matrix.

[0196] 12) Self-attention calculation:

[0197]

[0198] Where, d k It is the dimension of the key vector, used for scaling.

[0199] 13) Multi-head mechanism: Apply the attention mechanism to multiple subspaces and then concatenate the results.

[0200] 2) Feed-Forward Network:

[0201] 21) Two-layer fully connected network:

[0202] FFN(x)=ReLU(xW1+b1)W2+b2

[0203] Where W1, W2 and b1, b2 are learnable parameters.

[0204] 3) Residual connectivity and layer normalization:

[0205] 31) Residual Connections: Add residual connections after the attention layer and the feedforward network to alleviate the gradient vanishing problem.

[0206] 32) Layer normalization: Perform layer normalization on the results of residual connections to stabilize the training process.

[0207] 3. Stacking of Transformer layers:

[0208] 1) Layer setting: Stack multiple Transformer layers according to task requirements to enhance the expressive power of the model.

[0209] 4. Output characteristics:

[0210] 1) Global feature representation: After processing by the Transformer, the global feature F is obtained. Transformer It contains the dependencies between distant words in the sequence.

[0211] Step 7: Feature fusion and classification.

[0212] Local and global features are fused to form the final feature representation of the text, which is then input into a classifier to classify the text difficulty.

[0213] 1. Feature fusion:

[0214] 1) Method selection: Feature fusion can be performed by feature concatenation, weighted summation or attention mechanism.

[0215] 2) Example: F CNN and F Transformer Concatenate along the feature dimension:

[0216] F Final =[F CNN ;F Transformer ]

[0217] 2. Classifier Design:

[0218] 1) Classifier selection: Use Support Vector Machine (SVM), Multilayer Perceptron (MLP) and other machine learning models for classification.

[0219] 2) Input data: The fused features F Final Flatten it out and use it as input to the classifier.

[0220] 3. Training and Optimization:

[0221] 1) Loss function: Use the cross-entropy loss function or other suitable loss function to measure the difference between the predicted result and the true label.

[0222] 2) Optimization algorithm: Select a suitable optimization algorithm, such as Adam or SGD, and update the model parameters.

[0223] 4. Model Evaluation:

[0224] 1) Evaluation metrics: Use metrics such as accuracy, precision, recall, and F1-score to evaluate model performance.

[0225] 2) Cross-validation: k-fold cross-validation is used to evaluate the generalization ability of the model.

[0226] Through the above seven steps, this embodiment of the invention constructs a complete algorithm for Chinese text difficulty classification. This algorithm draws on the feature extraction technique of EEG Conformer, utilizes CNN to extract local features, capturing local patterns and adjacency relationships in word sequences; and utilizes Transformer to extract global features, capturing long-distance dependencies and global semantic information in the sequence. Finally, by fusing local and global features, the data is input into a classifier to classify text difficulty.

[0227] In this implementation, the CNN convolutional kernel size used in this invention is (6,1). A kernel of size 6 is used in the temporal direction to capture local three-word context features, while the kernel size in the channel direction is 1 to maintain feature stability along the channel dimension. The convolutional layers are followed by ReLU activation functions and batch normalization layers, and max pooling (poolSize=2) is used to downsample the output features. The Transformer part uses a two-layer stacked encoder, each layer containing an 8-head self-attention mechanism. The multi-head attention hidden dimension is set to 512, and the feedforward layer dimension is 2048. Positional encoding uses a sinusoidal positional encoding strategy.

[0228] In summary, this invention involves performing text segmentation on the acquired Chinese text to obtain a Chinese vocabulary set; obtaining the embedding representation of each Chinese word in the vocabulary set to obtain an embedding vector set; calculating linguistic indicators based on the embedding vector set; these linguistic indicators characterizing the lexical, sentence, discourse, and syntactic complexity features of the text; combining the embedding vector of each Chinese word with the linguistic indicators to construct a feature matrix; processing the feature matrix using a convolutional neural network to extract local features; these local features characterizing local patterns and relationships between adjacent elements in the data; processing the feature matrix using a pre-trained Transformer model to extract global features; these global features characterizing long-distance dependencies and global relationships in the word sequence; fusing the local and global features to form the final feature representation of the text; and inputting the final feature representation into a classifier to classify the text difficulty, obtaining the Chinese text difficulty classification result. This invention is applicable to various scenarios requiring evaluation of text readability, with high classification efficiency and accuracy.

[0229] The method provided by the embodiments of the present invention can be applied to the following fields:

[0230] 1. Government platform search: can enhance the accessibility of government data;

[0231] 2. Leveled reading: Able to customize reading materials according to skill levels;

[0232] 3. Test item development and evaluation: Able to create and evaluate educational effectiveness;

[0233] 4. Textbook compilation and optimization: can improve the quality of educational content;

[0234] 5. Reading skills training: It can develop and improve reading ability;

[0235] 6. Reading Disability Assistance: Provides support for individuals with reading difficulties;

[0236] 7. Library book classification: Effectively organizes library resources;

[0237] 8. Social Media Content Recommendation: Able to recommend relevant content on social media platforms.

[0238] Another aspect of this invention provides a Chinese text difficulty classification system, including:

[0239] The first module is used to perform text segmentation on the acquired Chinese text to be processed, and obtain a set of Chinese words.

[0240] The second module is used to obtain the embedding representation of each Chinese word in the Chinese word set, and to obtain the embedding vector set;

[0241] The third module is used to calculate linguistic indicators based on the Embedding vector set; the linguistic indicators are used to characterize the lexical, sentence, discourse, and syntactic complexity features of the text.

[0242] The fourth module is used to combine the embedding vector of each Chinese word with the linguistic indicators to construct a feature matrix;

[0243] The fifth module is used to process the feature matrix using a convolutional neural network to extract local features; these local features are used to characterize local patterns in the data and the relationships between adjacent elements.

[0244] The sixth module is used to process the feature matrix using a pre-trained Transformer model to extract global features; these global features are used to represent long-distance dependencies and global relationships in the word sequence.

[0245] The seventh module is used to fuse local and global features to form the final feature representation of the text; the final feature representation is then input into a classifier to classify the text difficulty, resulting in a Chinese text difficulty classification result.

[0246] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0247] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned Chinese text difficulty classification method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0248] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0249] Please see Figure 6 , Figure 6 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0250] The processor 601 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0251] The memory 602 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 602 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called and executed by the processor 601 using the Chinese text difficulty classification method of the embodiments of this invention.

[0252] The input / output interface 603 is used to implement information input and output;

[0253] The communication interface 604 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0254] Bus 605 transmits information between various components of the device (e.g., processor 601, memory 602, input / output interface 603, and communication interface 604);

[0255] The processor 601, memory 602, input / output interface 603, and communication interface 604 are connected to each other within the device via bus 605.

[0256] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described Chinese text difficulty classification method.

[0257] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0258] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0259] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.

[0260] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0261] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0262] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0263] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0264] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0265] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0266] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0267] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0268] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0269] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0270] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A method for classifying the difficulty of Chinese text, characterized in that, Includes the following steps: The acquired Chinese text to be processed is subjected to text segmentation to obtain a set of Chinese words; Obtain the embedding representation of each Chinese word in the Chinese vocabulary set to obtain an embedding vector set; Based on the Embedding vector set, linguistic indicators are calculated; these linguistic indicators are used to characterize the lexical, sentence, discourse, and syntactic complexity features of the text. The embedding vector of each Chinese word is combined with the linguistic indicators to construct a feature matrix; A convolutional neural network is used to process the feature matrix and extract local features; these local features are used to characterize local patterns in the data and the relationships between adjacent elements. The feature matrix is ​​processed using a pre-trained Transformer model to extract global features; The global features are used to characterize long-distance dependencies and global relationships in the word sequence; Local and global features are fused to form the final feature representation of the text; the final feature representation is then input into a classifier to classify the text difficulty, resulting in the Chinese text difficulty classification result.

2. The Chinese text difficulty classification method according to claim 1, characterized in that, The process of performing text segmentation on the acquired Chinese text to obtain a Chinese vocabulary set includes the following steps: Call the Kimi large model to select target word segmentation tool; Based on the word segmentation tool, the acquired Chinese text to be processed is segmented into words using a word segmentation strategy; After word segmentation is completed, stop words are filtered out. The words obtained from word segmentation are tagged with their parts of speech to obtain a set of Chinese vocabulary.

3. The Chinese text difficulty classification method according to claim 1, characterized in that, The step of obtaining the embedding representation of each Chinese word in the Chinese vocabulary set to obtain the embedding vector set includes the following steps: The pre-trained embedding model provides an embedding representation for each word with a dimension of 3072. For each word, call the API or interface of the Embedding model, input the word and get the corresponding Embedding vector.

4. The Chinese text difficulty classification method according to claim 1, characterized in that, The calculation of linguistic indicators based on the Embedding vector set includes the following steps: The average word frequency is determined by calculating the average frequency of occurrence of words and the ratio of high-frequency words to low-frequency words based on a word frequency dictionary or corpus; the lexical diversity is determined by calculating the type-tag ratio by dividing the number of different words by the total number of words; and then the lexical complexity index is determined based on the average word frequency and the lexical diversity. Sentence complexity metrics are determined by calculating average sentence length, number of subordinate clauses, and their proportion. The text structure indicators are determined by calculating the total number of words, the total number of sentences, and the keyword density. Syntactic complexity metrics are determined by calculating dependency tree depth and the number of syntactic clauses. The various indicators are combined into a feature vector; The linguistic indicators include at least average word frequency, type-to-tag ratio, average sentence length, clause ratio, topic word density, dependency tree depth, and number of syntactic clauses.

5. The Chinese text difficulty classification method according to claim 1, characterized in that, The step of combining the embedding vector of each Chinese word with the linguistic indicators to construct a feature matrix includes the following steps: For each word's embedding vector, 23 linguistic indicators are appended to form an extended vector; The feature matrix is ​​constructed by using each extended vector as the i-th column of the feature matrix.

6. The Chinese text difficulty classification method according to claim 1, characterized in that, The process of using a convolutional neural network to process the feature matrix and extract local features includes the following steps: By taking the feature matrix as input and adding a channel dimension, the corresponding input shape is determined. Temporal convolution of the convolutional layer performs one-dimensional convolution in the temporal dimension to capture local patterns in the word sequence, and channel convolution performs one-dimensional convolution in the channel dimension to capture the correlation between features. Constructing activation functions increases the model's non-linear representation capability, and adding batch normalization layers after convolutional layers stabilizes the training process and accelerates convergence; Constructing pooling layers involves using average pooling or max pooling layers to downsample features in both time and channel dimensions, and setting the pooling kernel size and stride as needed. After convolution and pooling layers, a feature tensor is obtained, and finally the feature is output; the feature tensor contains local patterns and correlation information between adjacent words; The expression for extracting local features is: F CNN =Pooling(Activation(Conv(E))), where Conv represents the convolution operation, Activation represents the activation function, and Pooling represents the pooling operation; The convolutional neural network has a kernel size of 6 in the time dimension to capture local features of three adjacent words; the activation function is ReLU, and the pooling layer is a max pooling layer with a kernel size of 2.

7. The Chinese text difficulty classification method according to claim 1, characterized in that, The process of using a pre-trained Transformer model to process the feature matrix and extract global features includes the following steps: Input data preparation specifically involves using features extracted by CNN as input to the Transformer and adding positional encoding to the input to determine the position of the words recognized by the model in the sequence. The Transformer architecture is constructed by building a multi-head self-attention mechanism, a feedforward neural network, residual connections, and layer normalization. Stacking Transformer layers involves stacking multiple Transformer layers according to task requirements to enhance the model's expressive power. After processing by the Transformer, global features are obtained; these global features contain the dependencies between distant words in the sequence. The Transformer uses a two-layer stacked Transformer encoder, with each layer including an eight-head self-attention mechanism and a feedforward neural network, and a hidden dimension of 512.

8. A Chinese text difficulty classification system, characterized in that, include: The first module is used to perform text segmentation on the acquired Chinese text to be processed, and obtain a set of Chinese words. The second module is used to obtain the embedding representation of each Chinese word in the Chinese word set, and to obtain the embedding vector set; The third module is used to calculate linguistic indicators based on the Embedding vector set; the linguistic indicators are used to characterize the lexical, sentence, discourse, and syntactic complexity features of the text. The fourth module is used to combine the embedding vector of each Chinese word with the linguistic indicators to construct a feature matrix; The fifth module is used to process the feature matrix using a convolutional neural network to extract local features; these local features are used to characterize local patterns in the data and the relationships between adjacent elements. The sixth module is used to process the feature matrix using a pre-trained Transformer model to extract global features; The global features are used to characterize long-distance dependencies and global relationships in the word sequence; The seventh module is used to fuse local and global features to form the final feature representation of the text; the final feature representation is then input into a classifier to classify the text difficulty, resulting in a Chinese text difficulty classification result.

9. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 7.

Images