Readability evaluation method based on language feature learner and contrastive information bottleneck

Abstract

The present invention belongs to the technical field of natural language processing, discloses a readability evaluation method based on a language feature learner and a contrast information bottleneck, establishes a readability evaluation model, and the readability evaluation model includes a language feature learner module, a deep feature extractor module and a contrast information bottleneck module; the pre-trained language feature learner module is used to learn the language feature representation of the text, the language feature representation is formed by the fusion of the deep language feature representations corresponding to each language feature, the deep feature extractor module is used to extract different levels of deep features from the text, and the contrast information bottleneck module is used to perform contrast learning from different levels of deep features to generate a deep representation that removes redundant information. The present invention enables a neural network branch to better extract the corresponding language features, and the obtained result simultaneously contains rich and complex language representations and fine deep representations, thereby improving the readability evaluation effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a readability evaluation method based on a language feature learner and a comparative information bottleneck. Background Art

[0002] Text readability assessment aims to evaluate the difficulty level of a text, and Automatic Readability Assessment (ARA) has been widely used in improving reading skills and promoting second language learning. In the early stages, ARA measured readability levels through empirical formulas based on some linguistic variables (such as text length). Although these readability formulas have good interpretability, they cannot capture complex language features, resulting in poor performance when processing complex texts. Subsequently, ARA models based on statistical learning rely on a large number of manually extracted language features (such as vocabulary, syntax, semantics, and discourse structure). These extracted language features can explicitly quantify the language properties of the text and are classified and evaluated by traditional machine learning models. However, due to the limited ability of the extracted language features to capture potential features such as context, these statistical learning models generally perform poorly.

[0003] With the development of deep learning, especially the emergence of pre-trained language models, researchers have proposed ARA models based on neural models, which are motivated by the powerful representation capabilities of deep neural networks. Although such neural models are effective, their performance is usually limited by the lack of training corpora. To alleviate this problem, a hybrid ARA model that integrates deep features and language features has attracted a lot of attention in the literature in recent years. Its motivation comes from previous studies that have proved the importance of language features for ARA. For example, pre-trained language models (such as BERT) are used to extract deep features, which are simultaneously input into statistical learning models together with manually extracted language features to obtain readability levels. However, existing hybrid ARA models mainly focus on shallow language features, have limited representation capabilities, and cannot capture complex language features. At the same time, existing hybrid ARA models mainly use orthogonal projection schemes to fuse deep features and language features. When multi-level deep features are involved, this method usually cannot effectively reduce the redundancy between deep features at various levels.

[0004] In the latest existing technologies, contrastive learning technology is introduced to enhance the extraction of deep representations, but the ARA model uses unsupervised contrastive learning technology. Since the redundant information in the training samples will reduce the training efficiency, this type of technology directly performs contrastive learning based on the similarity of text blocks. The object of contrastive learning does not involve classification results. Therefore, it is impossible to fully explore the characteristic differences in difficulty of texts of different difficulty levels, and the training effect of evaluating classification is therefore affected. Summary of the invention

[0005] The purpose of the present invention is to provide a readability evaluation method based on a language feature learner and a comparative information bottleneck, which is used to solve the technical problems that the prior art cannot fully explore the feature differences in difficulty of texts of different difficulty levels, and that redundant information exists in both training samples and deep features extracted therefrom, and these factors have an adverse effect on the training effect of evaluation classification.

[0006] The readability assessment method based on language feature learner and contrast information bottleneck establishes a readability assessment model, which includes a language feature learner module, a deep feature extractor module and a contrast information bottleneck module; the pre-trained language feature learner module is used to learn the language feature representation of the text, and the language feature representation is formed by fusion of deep language feature representations corresponding to each language feature, the deep feature extractor module is used to extract deep features of different levels from the text, and the contrast information bottleneck module is used to perform contrast learning from deep features of different levels to generate a deep representation that removes redundant information.

[0007] Preferably, the language feature learner module includes multiple language feature learners, which correspond one-to-one to the language features of the application, and each language feature learner includes a language feature branch and a neural network branch; the language feature branch generates relevant language feature values ​​based on the input text extraction and converts them into pseudo labels as supervisory information; the neural network branch is used to input the text into a pre-trained BigBird model to generate corresponding deep features, and then process it through a two-layer decoder to generate predicted pseudo labels, and pre-training of the corresponding language feature learner is achieved by simultaneously optimizing the cross-entropy loss of pseudo-label prediction and the cross-entropy loss of readability evaluation.

[0008] Preferably, for the i-th language feature learner, the text is firstly input into the language feature extractor and the pre-trained BigBird model in the language feature learner respectively; the language feature extractor outputs the corresponding i-th language feature learner in the language feature branch. i The language features are then transformed by the pseudo-label converter to obtain pseudo-labels as supervision information. y (i) ; At the same time, in the neural network branch, the pre-trained BigBird model outputs deep features, and the deep features are obtained through a two-layer decoder with the same architecture as in the contrast information bottleneck module to obtain relevant deep language feature representations, and the deep language feature representations are used as part of the features in the subsequent fusion stage; the deep language feature representations are provided as input to the nonlinear discriminator to generate predicted pseudo labels .

[0009] Preferably, in the two-layer decoder, the first layer decoder is a linear layer with Hardswish activation, and the second layer decoder is a linear layer; the nonlinear discriminator is a linear layer with softmax activation.

[0010] Preferably, the pseudo labels are generated by sorting the language feature values ​​of all samples and dividing the intervals into pseudo labels for the corresponding samples; the specific optimization method for pre-training the language feature learner module includes: after obtaining the deep language feature representation output by each language feature learner, all the deep language feature representations are fused through a mixed pooling operation and a linear layer to generate a final language feature representation, pre-training is performed through a readability evaluation task, and the language feature representation is input into a linear layer with softmax activation to output the evaluated readability level; based on the pre-training results, the pre-training loss of the language feature learner module The loss function is expressed as follows:

[0011] ,

[0012] in, k is the number of language feature learners, represents the cross entropy loss of the pseudo-label prediction associated with the i-th language feature, Represents the cross entropy loss for the readability evaluation of the language feature learner module.

[0013] Preferably, the deep feature extractor module extracts three levels of deep features, namely, document level, paragraph level and sentence level features, and the three levels of deep features are then spliced ​​into input features of the comparative information bottleneck module; in the comparative information bottleneck module, the input features are sequentially input into the refined neural network of the two-layer encoder to generate a refined deep representation of the text; the refined neural network has the same branch structure as the neural network in the language feature learner; subsequently, the text with the same readability level as the current text is set as a positive sample, and the remaining texts are set as negative samples, and the refined deep representations of different texts are subjected to comparative learning, and the refined deep representation obtained after the comparative learning is the final output deep representation.

[0014] Preferably, in the contrastive information bottleneck module, the refined depth representation is further input into a supervised neural network, and the supervised neural network has a structure symmetrical to the refined neural network; the refined depth representation is input into the supervised neural network to obtain a decoded representation as a self-supervisory label, thereby guiding the contrastive learning process of the input features through the self-supervisory label.

[0015] Preferably, the obtained deep representation and language feature representation are fused through an orthogonal projection layer, wherein the deep representation is used as a basis vector and the language feature representation is projected onto the deep representation to obtain a projection representation, which provides important supplementary information for readability assessment. The deep representation and the projection representation are then concatenated for readability assessment and input into a linear layer with softmax activation, which is used as a classifier to obtain the final evaluation result, i.e., the readability level.

[0016] Preferably, for the contrastive information bottleneck module, the relevant model loss includes the contrastive loss related to contrastive learning and the self-supervised information bottleneck loss; wherein the contrastive loss The loss function is as follows:

[0017] ,

[0018] Where N is the number of samples, is the set of readability levels, is the set of positive samples associated with the i-th readability level, Represents the positive sample set The base number, is the comparison sample set related to the i-th readability level, t is the temperature factor, which is an adjustable parameter used to adjust the contrast loss. a is a negative sample, p is a positive sample, z i Representation sample i The representation vector of z p Represents a positive sample p The representation vector of z a Represents negative samples a Representation vector of ; self-supervised information bottleneck loss The loss function is as follows:

[0019] ,

[0020] in, f represents the input features, f Representation as the decoded representation of the self-supervised label.

[0021] Preferably, the total loss of the readability assessment model The loss function is expressed as:

[0022] ,

[0023] in, The cross entropy loss for predicting readability ratings for the readability assessment model, is the contrast loss, is the self-supervised information bottleneck loss, l 1 and l 2 are two hyperparameters.

[0024] The present invention has the following advantages: the method uses a pre-trained language feature learner module to learn the language feature representation of the text, and determines the pseudo-label as the supervisory information in the language feature branch through the real language feature value, so that the pre-trained BigBird model can be supervised to learn the language features through this pseudo-label method, so that the neural network branch can better extract the corresponding language features, and the extraction result not only learns the representation of the language feature, but also has the properties of the deep feature. The structure of the two-layer decoder used in the neural network branch is also consistent with the decoder structure of the refined neural network in the contrast information bottleneck module that outputs the deep representation, so it can better realize the fusion of language feature representation and deep representation, and the result obtained contains rich and complex language representation and fine deep representation at the same time, thereby improving the effect of readability evaluation.

[0025] The present invention aims at the acquired deep features at the text level, paragraph level, and sentence level, and then combines the three levels to integrate contrastive learning into the information bottleneck principle. The contrastive learning is used to enhance the complementary relationship between the three levels of deep features in training, and to reduce the redundant information between the deep features at each level. In addition, a supervised neural network is introduced to realize the supervision of the contrastive learning process. In this way, the present invention can remove the feature-level redundant information of the same sample, and can also realize the mining of the difference information between samples of different categories, so that the readability evaluation classification of the present method can achieve very good results. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The flowchart of the readability evaluation method based on language feature learner and contrast information bottleneck of the present invention. DETAILED DESCRIPTION

[0027] The specific implementation modes of the present invention are further explained in detail below by describing the embodiments with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0028] like Figure 1As shown, the present invention provides a readability evaluation method based on a language feature learner and a contrastive information bottleneck, including: establishing a new readability evaluation model, the new readability evaluation model (LearnARA) includes a language feature learner (Linguistic Feature Learner, LFL) module, a deep feature extractor (Deep Feature Extractor, DFE) module and a contrastive Information Bottleneck (Contrastive Information Bottleneck, CIB) module; the pre-trained language feature learner module is used to learn the language feature representation of the text, the language feature representation is formed by fusion of deep language feature representations corresponding to each language feature, the deep feature extractor module is used to extract deep features of different levels from the text, and the contrastive information bottleneck module is used to perform contrastive learning from deep features of different levels to generate a deep representation that removes redundant information.

[0029] Language feature learner module: includes multiple language feature learners, each of which corresponds to the language features of the application one by one, and each language feature learner includes a language feature branch and a neural network branch; the language feature branch generates relevant language feature values ​​based on the input text extraction and converts them into pseudo labels as supervision information; the neural network branch is used to input the text into the pre-trained BigBird model to generate corresponding deep features, and then process it through a two-layer decoder to generate predicted pseudo labels, and pre-training of the corresponding language feature learner is achieved by simultaneously optimizing the cross-entropy loss of pseudo-label prediction and the cross-entropy loss of readability evaluation.

[0030] Specifically, for the i-th language feature learner, the text is first input into the language feature extractor and the pre-trained BigBird model in the language feature learner respectively; in the language feature branch, the language feature extractor outputs the corresponding i-th language feature, such as text length, which is then converted by the pseudo-label converter to obtain a pseudo-label as supervision information. y (i) ; At the same time, in the neural network branch, the pre-trained BigBird model outputs deep features, which are used to obtain relevant deep language feature representations through a two-layer decoder with the same architecture as in the contrastive information bottleneck (CIB) module. The deep language feature representation is used as part of the features in the subsequent fusion stage; the deep language feature representation is provided as input to the nonlinear discriminator to generate predicted pseudo-labels . The language feature learner minimizes the cross entropy loss of pseudo label prediction and the cross entropy loss of readability evaluation at the same time, thereby optimizing the i-th language feature learner. In the two-layer decoder, the first layer decoder is a linear layer with Hardswish activation, and the second layer decoder is a linear layer; the nonlinear discriminator is a linear layer with softmax activation. Since the neural network in the language feature learner and the contrastive information bottleneck module uses the same two-layer decoder for decoding, the results can be better integrated.

[0031] In the above pseudo-label converter, pseudo-labels are generated by sorting the language feature values ​​of all samples and dividing them into intervals as pseudo-labels for the corresponding samples. For example, there are 1,000 texts, and the number of nouns in them is extracted and sorted as "1, 3, 4...140". We divide this sequence into several intervals of categories, such as 5. Then the pseudo-labels of samples 1-200 are 0, 201-400 are 1, and so on. Pseudo-labels are not the output predicted by the model, but the result of the real language feature value. In this way, the pre-trained BigBird model can be supervised to learn language features through this pseudo-label method. By designing multiple groups of such language feature learners, this method not only learns the representation of the language feature, but also the deep features output by the BigBird model, which goes beyond the representation of the language feature and has the properties of both the representation of the language feature and the deep feature.

[0032] The specific optimization method for pre-training the language feature learner module includes: after obtaining the deep language feature representation output by each language feature learner, all the deep language feature representations are fused through a mixed pooling operation and a linear layer to generate the final language feature representation. In order to further supervise the learning of language representation, this method is pre-trained through the readability evaluation task, and the language feature representation is input into a linear layer with softmax activation to output the evaluated readability level. Based on the pre-training results, the pre-training loss of the language feature learner module is The loss function is expressed as follows:

[0033] ,

[0034] Where k is the number of language feature learners, represents the cross entropy loss of the pseudo-label prediction associated with the i-th language feature (corresponding to the i-th language feature extractor), Represents the cross entropy loss for the readability evaluation of the language feature learner module.

[0035] Deep Feature Extractor Module: This module extracts three levels of deep features, namely document-level, paragraph-level, and sentence-level features, to fully explore the semantic, structural, and contextual information that are crucial for readability assessment; by utilizing these three levels of features, the proposed model is able to handle texts of any length. Specifically, based on the superiority of the BigBird model in extracting features from long documents, the deep feature extractor module takes the entire text as input to the pre-trained BigBird model to generate document-level features; at the same time, the deep feature extractor module inputs the paragraphs and sentences obtained by dividing the text into the pre-trained BigBird model to obtain the corresponding deep features. The deep features generated by each paragraph are fused through a linear layer with a mixed pooling operation to generate a paragraph-level feature, and the deep features generated by each sentence are fused through a linear layer with a mixed pooling operation to generate a sentence-level feature. The three levels of deep features, namely, document-level, paragraph-level, and sentence-level features, are then concatenated into the input features of the comparative information bottleneck module.

[0036] Contrastive Information Bottleneck Module: The input features include three different levels of deep features extracted from the same text at the document level, paragraph level, and sentence level. They are essentially complementary, but also contain a lot of redundancy. In order to enhance the complementarity of these deep features and reduce redundancy, this method introduces a contrastive information bottleneck module, integrating contrastive learning into the information bottleneck principle.

[0037] Specifically, in the contrastive information bottleneck module, the input features are sequentially input into the refinement neural network of the two-layer encoder to generate a refined deep representation of the text; the refinement neural network has the same branch structure as the neural network in the language feature learner, the first layer encoder is a linear layer with Hardswish activation, and the second layer decoder is a linear layer. Subsequently, the text with the same readability level as the current text is set as a positive sample, and the remaining texts are negative samples, and the refined deep representations of different texts are contrastively learned. Inspired by the information bottleneck principle, the method further inputs the refined deep representation into a supervised neural network, and the supervised neural network has a structure symmetrical to the refined neural network. In the supervised neural network, the first layer encoder is a linear layer, and the second layer decoder is a linear layer with Hardswish activation; the refined deep representation is input into the supervised neural network to obtain a decoded representation as a self-supervisory label, thereby guiding the contrastive learning process of the input features through the self-supervisory label, and the refined deep representation obtained after the contrastive learning is completed is the final output deep representation.

[0038] For the contrastive information bottleneck module, the relevant model losses include the contrastive loss related to contrastive learning and the self-supervised information bottleneck loss.

[0039] Among them, contrast loss The loss function is as follows:

[0040] ,

[0041] Where N is the number of samples, is the set of readability levels, is the set of positive samples associated with the i-th readability level, Represents the positive sample set The base number, is the comparison sample set related to the i-th readability level, t is the temperature factor, which is an adjustable parameter used to adjust the contrast loss. a is a negative sample, p is a positive sample, z i Representation sample i The representation vector of z p Represents a positive sample p The representation vector of z a Represents negative samples a The representation vector of .

[0042] Self-supervised information bottleneck loss The loss function is as follows:

[0043] ,

[0044] in, f represents the input features (i.e. the result of concatenating three levels of deep features), f Represents the decoded representation of the self-supervised label. For computational efficiency considerations, this method uses L2 loss to measure the input features f and the decoded representation as self-supervised label f The distance between.

[0045] Finally, the obtained deep representation and language feature representation are fused through an orthogonal projection layer, in which the deep representation is used as the basis vector and the language feature representation is projected onto the deep representation to obtain the projection representation, which provides important supplementary information for readability evaluation. In order to further enhance the representation information, this method concatenates the deep representation and the projection representation for readability evaluation, inputs a linear layer with softmax activation, and uses the linear layer as a classifier to obtain the final evaluation result, i.e., the readability level.

[0046] The total loss of the model used in this method The loss function is expressed as:

[0047] ,

[0048] in, The cross entropy loss for predicting readability ratings for the readability assessment model, is the contrast loss, is the self-supervised information bottleneck loss, l 1 and l 2 are two hyperparameters.

[0049] The present invention is described above by way of example in conjunction with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned method. As long as various non-substantial improvements are made using the inventive concept and technical solution of the present invention, or the inventive concept and technical solution are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A readability evaluation method based on a language feature learner and contrastive information bottleneck, characterized by: Establishing a readability evaluation model, wherein the readability evaluation model includes a language feature learner module, a deep feature extractor module, and a comparative information bottleneck module; The language feature learner module includes a plurality of language feature learners, each of which corresponds to the language features of the application, and each of which includes a language feature branch and a neural network branch; the language feature branch generates relevant language feature values ​​based on the input text extraction and converts them into pseudo labels as supervisory information; the neural network branch is used to input the text into a pre-trained BigBird model to generate corresponding deep features, and then process the text through a two-layer decoder to generate predicted pseudo labels, and pre-train the corresponding language feature learner by simultaneously optimizing the cross entropy loss of pseudo label prediction and the cross entropy loss of readability evaluation; The deep feature extractor module extracts three levels of deep features at the document level, paragraph level and sentence level, i.e., document level, paragraph level and sentence level features, and the three levels of deep features are then spliced ​​into input features of the comparative information bottleneck module; in the comparative information bottleneck module, the input features are sequentially input into the refined neural network of the two-layer encoder to generate a refined deep representation of the text; the refined neural network has the same branch structure as the neural network in the language feature learner; then, the text with the same readability level as the current text is set as a positive sample, and the remaining texts are negative samples, and the refined deep representations of different texts are subjected to comparative learning, and the refined deep representation obtained after the comparative learning is the final output deep representation; The obtained deep representation and language feature representation are fused through an orthogonal projection layer, in which the deep representation is used as the basis vector and the language feature representation is projected onto the deep representation to obtain the projection representation, which provides important supplementary information for readability assessment. The deep representation and the projection representation are then concatenated for readability assessment and input into a linear layer with softmax activation, which is used as a classifier to obtain the final evaluation result, i.e., the readability level.

2. The readability evaluation method based on language feature learner and contrast information bottleneck according to claim 1, characterized in that: For the i-th language feature learner, the text is first input into the language feature extractor and the pre-trained BigBird model in the language feature learner respectively; in the language feature branch, the language feature extractor outputs the corresponding i-th i The language features are then transformed by the pseudo-label converter to obtain pseudo-labels as supervision information. y (i) ; At the same time, in the neural network branch, the pre-trained BigBird model outputs deep features, and the deep features are obtained through a two-layer decoder with the same architecture as in the contrast information bottleneck module to obtain relevant deep language feature representations, and the deep language feature representations are used as part of the features in the subsequent fusion stage; the deep language feature representations are provided as input to the nonlinear discriminator to generate predicted pseudo labels .

3. The readability evaluation method based on language feature learner and contrast information bottleneck according to claim 2 is characterized in that: In the two-layer decoder, the first layer decoder is a linear layer with Hardswish activation, and the second layer decoder is a linear layer; The nonlinear discriminator is a linear layer with softmax activation.

4. The readability evaluation method based on language feature learner and contrast information bottleneck according to claim 2 or 3, characterized in that: The pseudo-labels are generated by sorting the language feature values ​​of all samples and dividing the intervals into pseudo-labels for the corresponding samples. The specific optimization method for pre-training the language feature learner module includes: after obtaining the deep language feature representation output by each language feature learner, all the deep language feature representations are fused through a mixed pooling operation and a linear layer to generate the final language feature representation, and pre-training is performed through the readability evaluation task, and the language feature representation is input into a linear layer with softmax activation to output the evaluated readability level; based on the pre-training results, the pre-training loss of the language feature learner module The loss function is expressed as follows: ， in, k is the number of language feature learners, represents the cross entropy loss of the pseudo-label prediction associated with the i-th language feature, Represents the cross entropy loss for the readability evaluation of the language feature learner module.

5. The readability evaluation method based on language feature learner and contrast information bottleneck according to claim 4 is characterized in that: In the contrastive information bottleneck module, the refined deep representation is further input into a supervised neural network, which has a structure symmetrical to the refined neural network; the refined deep representation is input into the supervised neural network to obtain a decoded representation as a self-supervisory label, thereby guiding the contrastive learning process of the input features through the self-supervisory label.

6. The readability evaluation method based on language feature learner and contrast information bottleneck according to claim 5, characterized in that: For the contrastive information bottleneck module, the relevant model losses include the contrastive loss related to contrastive learning and the self-supervised information bottleneck loss; among them, the contrastive loss The loss function is as follows: ， Where N is the number of samples, is the set of readability levels, is the set of positive samples associated with the i-th readability level, Represents the positive sample set The base number, is the comparison sample set related to the i-th readability level, τ is the temperature factor, which is an adjustable parameter used to adjust the contrast loss. a is a negative sample, p is a positive sample, z i Representation sample i The representation vector of z p Represents a positive sample p The representation vector of z a Represents negative samples a Representation vector of ; self-supervised information bottleneck loss The loss function is as follows: ， in, f represents the input features, f ̃ Representation as the decoded representation of the self-supervised label.

7. The readability evaluation method based on language feature learner and contrast information bottleneck according to claim 6 is characterized by: The total loss of the readability evaluation model The loss function is expressed as: ， in, The cross entropy loss for predicting readability ratings for the readability assessment model, is the contrast loss, is the self-supervised information bottleneck loss, λ 1 and λ 2 are two hyperparameters.

Images