Chinese character sentence classification method based on semantic information enhancement

Abstract

The present invention specifically relates to a Chinese character sentence classification method based on semantic information enhancement, including: inputting the Chinese character sentence to be classified into a trained classification model, and outputting the corresponding predicted classification label; during training: firstly constructing a discrimination dictionary associated with radicals based on a training data set; secondly, converting the Chinese character sentences in the training data set into simplified and traditional Chinese characters to obtain simplified and traditional Chinese character sentences; then extracting the classification representation and contextual representation of the Chinese character sentences; at the same time, determining the keywords in the Chinese character sentences in combination with the discrimination dictionary, and assigning attention weights to each keyword in combination with an attention mechanism, and then calculating the final representation of the simplified and traditional texts; finally, generating predicted classification labels based on the final representations of the simplified and traditional texts respectively and fusing them to obtain the overall predicted classification labels to optimize the model parameters of the classification model. The present invention can effectively consider the semantic loss of Chinese characters in the process of Chinese character simplification, and eliminate the interference of Chinese characters with low discrimination on sentences.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a Chinese character sentence classification method based on semantic information enhancement. Background Art

[0002] In recent years, deep learning has made great progress in various fields such as images, audio, and text. In the field of text, the introduction of the pre-trained model BERT is a groundbreaking work. Based on BERT, various improved models such as ALBERT and RoBERTa have been born, but most of these models are trained on phonetic texts such as English. It is precisely because most models were originally designed for English that the research on Chinese sentences is not yet fully mature in current model research.

[0003] From Cangjie's creation of characters to the simplified characters currently in use, Chinese characters have evolved step by step. Unlike Latin, English and other phonetic scripts, Chinese characters are non-phonetic characters, and their pictographic structure contains more information. In order to better complete the processing of Chinese character sentences, relevant research in the field of Chinese language is developing rapidly. In terms of pre-training models, Baidu's Chinese pre-training model ernie is being widely used in his tasks. For Chinese characters, which are pictographic characters, there are also some solutions that apply this structure. For example, "Improving the Calculation Method of Chinese Character Glyph Similarity" uses the knowledge representation of radicals. Shannon Technology's pre-training model ChineseBERT that integrates glyph and phonetic information, and Tao's RAM model for the characteristics of Chinese characters' radicals, all integrate the structural characteristics of Chinese characters into the information of deep learning models.

[0004] The applicant found that in order to facilitate the use and learning of the public, eliminate illiteracy and popularize education more quickly, some different traditional Chinese characters were simplified into the same simplified Chinese characters. In the process of simplifying traditional characters into simplified characters, the original prior knowledge and concepts of Chinese characters were lost, which led to the problem of semantic loss of Chinese characters. At the same time, there are many common Chinese characters in Chinese sentences, such as you, me, him, etc. These Chinese characters appear very frequently, but these Chinese characters have low discrimination and low influence on the overall sentence. For example, if there is a "he" character in every sentence, then this "he" character cannot effectively contribute to the accuracy of classification. These Chinese characters with low discrimination interfere with the overall sentence. Therefore, how to design a Chinese sentence classification method that can effectively consider the semantic loss of Chinese characters in the process of Chinese character simplification and can eliminate the interference of Chinese characters with low discrimination on sentences is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the deficiencies of the above-mentioned prior art, the technical problem to be solved by the present invention is: how to provide a Chinese character sentence classification method based on semantic information enhancement, so as to effectively consider the semantic loss of Chinese characters in the process of Chinese character simplification, and eliminate the interference of Chinese characters with low discrimination on sentences, and then integrate the Chinese character structure to enhance the semantic information of Chinese character sentences, thereby improving the accuracy and practicality of Chinese character sentence classification.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] The Chinese character sentence classification method based on semantic information enhancement includes:

[0008] S1: Obtain the Chinese character sentence to be classified;

[0009] S2: Input the Chinese character sentence to be classified into the trained classification model and output the corresponding predicted classification label;

[0010] When training the classification model, first, a discrimination dictionary associated with radicals is constructed based on the training data set; secondly, simplified and traditional Chinese characters are converted into Chinese characters in the training data set to obtain simplified and traditional Chinese character sentences; then, the classification representation and contextual representation of simplified and traditional Chinese character sentences are extracted; at the same time, the discrimination dictionary is combined to determine the keywords in the Chinese character sentences, and the attention mechanism is combined to assign attention weights to each keyword, and then the classification representation and contextual representation are combined to calculate the final representation of the simplified and traditional texts; finally, the predicted classification labels are generated based on the final representations of the simplified and traditional texts, respectively, and then the predicted classification labels of the simplified and traditional texts are combined to obtain the overall predicted classification labels, and the model parameters of the classification model are optimized based on the overall predicted classification labels;

[0011] S3: Generate the classification result of the Chinese character sentence to be classified based on the predicted classification label output by the classification model.

[0012] Preferably, in step S2, a discrimination dictionary is constructed by the following steps:

[0013] S201: For each Chinese character sentence in the training data set: first, obtain the number of occurrences of each radical in the corresponding Chinese character sentence; then select radicals in descending order of the number of occurrences, and calculate the cosine similarity between the Chinese character corresponding to each radical and the corresponding Chinese character sentence; finally, generate a Chinese character similarity set for each radical;

[0014] S202: merging the Chinese character similarity sets corresponding to the radicals in each Chinese character sentence to form a similarity matrix corresponding to the radicals; then calculating the variance of all cosine similarities of the corresponding Chinese characters in the similarity matrix corresponding to the radicals as the discrimination of the Chinese character;

[0015] S203: for each Chinese character sentence: arranging the discrimination degrees of each Chinese character corresponding to each radical in descending order to obtain a discrimination degree descending list of each radical, and then integrating the discrimination degree descending lists of each radical to obtain a discrimination degree descending list of the Chinese character sentence;

[0016] S204: Accumulate the descending list of discrimination of each Chinese character sentence, and arrange the discrimination of each Chinese character corresponding to each radical in descending order to obtain a discrimination dictionary of each radical.

[0017] Preferably, in step S201, the similarity matrix is ​​calculated by the following formula:

[0018] C Rn =cos_sim(set R (S n )×set(R));

[0019] Where: C Rn Represents the similarity matrix of radical R; cos_sim represents the calculation of cosine similarity; set R (S n ) represents the Chinese sentence S n set(R) represents the set of Chinese characters consisting of radical R in the training data set S.

[0020] Preferably, in step S204, the discrimination dictionary is expressed by the following formula:

[0021]

[0022] L Rni,1 βC Rni,1 ;

[0023]

[0024] Where: L R represents the discrimination dictionary of radical R; n represents the number of Chinese character sentences in the training data set; L Ri represents the descending list of the discrimination of the radical R in the i-th Chinese character sentence. The descending list of discrimination is an array with two columns. The first column is the Chinese character and the second column is the discrimination of the corresponding Chinese character. Rni,1 Indicates the first column of the ith row in the descending order of discrimination, L Rni,2 Represents the second column of the ith row of the descending list of discrimination; C Rni represents the cosine similarity between the ith Chinese character in the nth Chinese character sentence and the Chinese character sentence; C Rni,j represents the cosine similarity between the ith Chinese character in the nth Chinese character sentence and the jth Chinese character in the Chinese character set set(R); m represents the number of Chinese characters in the Chinese character set set(R).

[0025] Preferably, in step S2, the final text representation is generated by the following steps:

[0026] S211: determining keywords in the Chinese character sentence based on the discrimination dictionary and generating a keyword matrix;

[0027] S212: inputting the Chinese sentence into the pre-trained BERT model, outputting the classification representation and text representation of the Chinese sentence; then inputting the text representation into the BiLSTM model to generate the contextual representation of the Chinese sentence;

[0028] S213: Using the keyword matrix as the key and value of the attention module, and then using the classification representation and context representation of the Chinese character sentence as the query of the attention module respectively;

[0029] S214: Match each Chinese character vector in the query to the keyword in the keyword matrix one by one; then calculate the attention weight of each keyword, and then combine the classification representation and the context representation to calculate the corresponding classification attention representation and context attention representation respectively;

[0030] S215: Concatenate the classification attention representation, context attention representation, classification representation and context representation of the Chinese character sentence to obtain the final text representation of the Chinese character sentence.

[0031] Preferably, in step S211, a keyword matrix is ​​generated by the following steps:

[0032] S2111: extracting radical information of the Chinese character sentence to determine a target radical;

[0033] S2112: extracting a target keyword of a target radical from a discrimination dictionary, and then selecting corresponding keywords from the Chinese character sentence based on the target keyword to form a keyword matrix.

[0034] Preferably, in step S2111, the radical information of the Chinese character sentence is obtained by the following formula:

[0035]

[0036] wr i βRadical_Query(w i );

[0037] Where: dict(S n ) represents the Chinese sentence S n Radical information of ;w i Represents a Chinese sentence S n Chinese characters in wr i Represents Chinese character w iRadical_Query represents a pre-built radical dictionary.

[0038] Preferably, in step S2112, the keyword matrix is ​​calculated by the following formula:

[0039]

[0040] Where: D represents the keyword matrix; L R represents the discrimination dictionary; set(R) represents the set of Chinese characters consisting of radical R in the training data set S.

[0041] Preferably, in step S212, the classification representation and text representation are extracted by the following formula:

[0042] C s ,E s βBERT (Sentence);

[0043] Where: C s Indicates classification representation; E s Indicates text representation; Sentence indicates a Chinese sentence;

[0044] The context representation is calculated by the following formula:

[0045] Y i ={y1,y2,...,y m};

[0046]

[0047]

[0048]

[0049] Where: Y i represents the context representation of the i-th Chinese sentence; yi represents the i-th character in the Chinese sentence; s i It represents the previous legacy information obtained by the i-th Chinese character sentence through the input gate and forget gate of LSTM; Indicates forward calculation; Indicates backward calculation; Indicates that the i-1th Chinese character of the Chinese sentence is combined with the corresponding previous remaining information s i The information of the i-th Chinese character is obtained by backward calculation.

[0050] Preferably, in step S214, the attention weight is calculated by the following formula:

[0051]

[0052] Where: αi Represents a Chinese sentence S n The classification representation C s Or the context indicates Y s The attention weight of the i-th Chinese character vector in queryi; queryi represents the i-th Chinese character vector in query, and query is the Chinese character sentence S n The classification representation C s Or the context indicates Y s ;e r Represents the keyword corresponding to the i-th Chinese character vector in the keyword matrix;

[0053] The classification attention representation and context attention representation are calculated by the following formula:

[0054]

[0055]

[0056] Where: C′ s , Y′ s Represents Chinese character sentence S n The classification attention representation and contextual attention representation of α i Represents a Chinese sentence S n The classification representation C s Or the context indicates Y s The attention weight of the i-th Chinese character vector in e; r Represents the keyword corresponding to the i-th Chinese character vector in the keyword matrix.

[0057] Compared with the prior art, the Chinese character sentence classification method based on semantic information enhancement in the present invention has the following beneficial effects:

[0058] The present invention obtains simplified and traditional Chinese character sentences by converting simplified and traditional Chinese characters, and then predicts and classifies and trains classification models according to the semantic representations (classification representation and context representation) of simplified and traditional Chinese character sentences respectively. This method of combining traditional and simplified Chinese character sentences can well integrate the word formation method of Chinese characters and the evolution of Chinese characters, can effectively consider the semantic loss of Chinese characters in the process of Chinese character simplification, and then integrate the Chinese character structure to enhance the semantic information of Chinese character sentences, thereby improving the accuracy of Chinese character sentence classification. At the same time, the present invention pre-constructs a discrimination dictionary associated with radicals according to a training data set, and then determines the keywords in the Chinese character sentences in combination with the discrimination dictionary in the process of generating the semantic representation of Chinese character sentences, and combines the attention mechanism to assign attention weights to each keyword, so that the semantic representation of Chinese character sentences can be further enhanced through keywords and their attention weights, and the interference of Chinese characters with low discrimination on sentences can be eliminated by combining the radical information of Chinese characters, which can also enhance the semantic information of Chinese character sentences, thereby further improving the practicality of Chinese character sentence classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0060] Figure 1 It is a logical block diagram of the Chinese character sentence classification method based on semantic information enhancement;

[0061] Figure 2 This is the network structure diagram of the classification model;

[0062] Figure 3 A diagram of the network structure for constructing a discriminability dictionary;

[0063] Figure 4 This is the simplified-traditional conversion judgment tree. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0065] It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. In the description of the present invention, it should be noted that the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the invention product is usually placed when used, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance. In addition, the terms "horizontal", "vertical", etc. do not mean that the components are required to be absolutely horizontal or suspended, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted. In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0066] The following is a further detailed description through specific implementation methods:

[0067] Example:

[0068] This embodiment discloses a Chinese character sentence classification method based on semantic information enhancement.

[0069] like Figure 1 As shown, the Chinese character sentence classification method based on semantic information enhancement includes:

[0070] S1: Obtain the Chinese character sentence to be classified;

[0071] S2: Input the Chinese character sentence to be classified into the trained classification model and output the corresponding predicted classification label;

[0072] like Figure 2As shown, during training: first, a discrimination dictionary associated with radicals is constructed based on the training data set; secondly, simplified and traditional Chinese characters are converted into Chinese characters in the training data set to obtain simplified and traditional Chinese character sentences; then, the classification representation and contextual representation of simplified and traditional Chinese character sentences are extracted; at the same time, the discrimination dictionary is combined to determine the keywords in the Chinese character sentences, and the attention mechanism is combined to assign attention weights to each keyword, and then the classification representation and contextual representation are combined to calculate the final representation of the simplified and traditional texts; finally, the predicted classification labels are generated based on the final representations of the simplified and traditional texts, respectively, and then the predicted classification labels of the simplified and traditional texts are integrated to obtain the overall predicted classification labels, and the model parameters of the classification model are optimized based on the overall predicted classification labels;

[0073] In this embodiment, the final representation of the text is used as the input of the fully connected layer to generate the corresponding predicted classification label; the fully connected layer uses the softmax function as the activation function. In addition, the corresponding cross entropy loss is calculated based on the overall predicted classification label and the corresponding true label, and then back propagation is performed through the cross entropy loss to optimize the model parameters of the classification model.

[0074] S3: Generate the classification result of the Chinese character sentence to be classified based on the predicted classification label output by the classification model.

[0075] In this embodiment, the predicted classification label refers to the type probability distribution of the Chinese sentence, and the type of the Chinese sentence can be judged according to the type probability distribution of the Chinese sentence, such as judging which category the news belongs to, judging whether an email is spam, etc.

[0076] The present invention obtains simplified and traditional Chinese character sentences by converting simplified and traditional Chinese characters, and then predicts and classifies and trains classification models according to the semantic representations (classification representation and context representation) of simplified and traditional Chinese character sentences respectively. This method of combining traditional and simplified Chinese character sentences can well integrate the word formation method of Chinese characters and the evolution of Chinese characters, can effectively consider the semantic loss of Chinese characters in the process of Chinese character simplification, and then integrate the Chinese character structure to enhance the semantic information of Chinese character sentences, thereby improving the accuracy of Chinese character sentence classification. At the same time, the present invention pre-constructs a discrimination dictionary associated with radicals according to a training data set, and then determines the keywords in the Chinese character sentences in combination with the discrimination dictionary in the process of generating the semantic representation of Chinese character sentences, and combines the attention mechanism to assign attention weights to each keyword, so that the semantic representation of Chinese character sentences can be further enhanced through keywords and their attention weights, and the interference of Chinese characters with low discrimination on sentences can be eliminated by combining the radical information of Chinese characters, which can also enhance the semantic information of Chinese character sentences, thereby further improving the practicality of Chinese character sentence classification.

[0077] In the specific implementation process, Figure 3 As shown, the discrimination dictionary is constructed through the following steps:

[0078] S201: For each Chinese character sentence in the training dataset: First, obtain the occurrence times of each radical in the corresponding Chinese character sentence; then, select the radicals in descending order of occurrence times, and calculate the cosine similarity between the Chinese characters corresponding to each radical and the corresponding Chinese character sentence; finally, generate the Chinese character similarity set for each radical.

[0079] S202: Integrate the Chinese character similarity sets of the corresponding radicals in each Chinese character sentence to form the similarity matrix of the corresponding radical; then, calculate the variance of all cosine similarities of the corresponding Chinese characters in the similarity matrix of the corresponding radical as the discrimination degree of the Chinese character.

[0080] S203: For each Chinese character sentence: Sort the discrimination degrees of the Chinese characters corresponding to each radical in descending order to obtain the discrimination degree descending list of each radical, and then integrate the discrimination degree descending lists of each radical to obtain the discrimination degree descending list of the Chinese character sentence.

[0081] S204: Accumulate the discrimination degree descending lists of each Chinese character sentence, and sort the discrimination degrees of the Chinese characters corresponding to each radical in descending order to obtain the discrimination degree dictionary of each radical.

[0082] For example, in a certain Chinese character sentence, "jiang", "he", "hu", and "hai" appear, and the radical "氵" appears with the highest frequency.

[0083] First, calculate to obtain {氵: (similarity a1, similarity b1, similarity b1, similarity d1)}. Among them, a, b, c, and d are the cosine similarities of the Chinese characters "jiang", "he", "hu", and "hai" corresponding to the radical "氵" with respect to the Chinese character sentence.

[0084] Second, since there are a large number of Chinese character sentences in the training dataset, there may be {氵: (similarity a2, similarity b2, similarity b2)} or {氵: (similarity a3, similarity b3)} in other Chinese character sentences, and the similarities a1, a2, and a3 may be different.

[0085] Integrate the Chinese character similarity sets of the radical "氵" in each Chinese character sentence to form the similarity matrix of the corresponding radical "氵": {氵: ("jiang":

a1, a2, a3,...

b1, g2, g3,...

[0086] Third, calculate the variance of all cosine similarities (a1, a2, a3,...) of the Chinese character "jiang" corresponding to the radical "氵" as the discrimination degree of the Chinese character "jiang", such as 0.5; similarly, calculate the discrimination degrees of the Chinese characters "he", "hu", and "hai" as 0.6, 0.4, and 0.7.

[0087] At this time, a descending list of discrimination degrees corresponding to the Chinese character sentence for the radical "氵" is obtained: {氵: "海": 0.7; "河": 0.6; "江": 0.5; "海": 0.4}, and then a descending list of discrimination degrees for the corresponding Chinese character sentence is obtained: {氵: ("海": 0.7; "河": 0.6; "江": 0.5; "海": 0.4); 辶: ("这": 0.7; "过": 0.3);...}.

[0088] Finally, the descending lists of discrimination degrees of each Chinese character sentence are accumulated, and the discrimination degrees of each Chinese character corresponding to each radical are sorted in descending order to obtain the discrimination degree dictionary of each radical. For example, for the radical "氵": {氵: "汤": 0.9; "汗": 0.8; "污": 0.75; "海": 0.7; "池": 0.7;...}.

[0089] The similarity matrix is calculated through the following formula:

[0090] C Rn = cos_sim(set R (S n )) × set(R));

[0091] In the formula: C Rn represents the similarity matrix of the radical R; cos_sim represents calculating the cosine similarity; set R (S n ) represents the set of Chinese characters composed of the radical R in the Chinese character sentence S n ; set(R) represents the set of Chinese characters composed of the radical R in the training data set S.

[0092] When calculating the cosine similarity, the Chinese characters in set R (S n ) and set R (S n ) can be used as the input of the FastText model, and then the cosine similarity is calculated based on the training vectors output by the FastText model. Among them, FastText is a text classification tool launched by Facebook AI Research. It can also be used to train word vectors, sentence vectors, etc. Its biggest feature is that the model is simple and the training speed is fast. For each text, the first step is to vectorize all words and use them as input; the second step is to average all the input vectors in the hidden layer to obtain a new vector; the third step outputs the prediction value.

[0093] FastText uses the character-level n-gram method to perform embedding on word segmentation in the paper. For example, for the word apple, it will be divided into ap / ppl / ple and perform embedding, and finally sum to obtain the embedding result of apple.

[0094] The discrimination dictionary is expressed by the following formula:

[0095]

[0096] L Rni,1 =C Rni,1 ;

[0097]

[0098] Where: L R represents the discrimination dictionary; n represents the number of Chinese character sentences in the training data set; L Ri represents the descending list of the discrimination of the radical R in the i-th Chinese character sentence. The descending list of discrimination is an array with two columns. The first column is the Chinese character and the second column is the discrimination of the corresponding Chinese character. Rni,1 Indicates the first column of the ith row in the descending order of discrimination, L Rni,2 Represents the second column of the ith row of the descending list of discrimination; C Rni represents the cosine similarity between the ith Chinese character in the nth Chinese character sentence and the Chinese character sentence; C Rni,j represents the cosine similarity between the ith Chinese character in the nth Chinese character sentence and the jth Chinese character in the Chinese character set set(R); m represents the number of Chinese characters in the Chinese character set set(R).

[0099] In the specific implementation process, the final text representation is generated through the following steps:

[0100] S211: determining keywords in the Chinese character sentence based on the discrimination dictionary and generating a keyword matrix;

[0101] S212: inputting the Chinese sentence into the pre-trained BERT model, outputting the classification representation and text representation of the Chinese sentence; then inputting the text representation into the BiLSTM model to generate the contextual representation of the Chinese sentence;

[0102] S213: Using the keyword matrix as the key and value of the attention module, and then using the classification representation and context representation of the Chinese character sentence as the query of the attention module respectively;

[0103] S214: Match each Chinese character vector in the query to the keyword in the keyword matrix one by one; then calculate the attention weight of each keyword, and then combine the classification representation and the context representation to calculate the corresponding classification attention representation and context attention representation respectively;

[0104] S215: Concatenate the classification attention representation, context attention representation, classification representation and context representation of the Chinese character sentence to obtain the final text representation of the Chinese character sentence.

[0105] The keyword matrix is ​​generated through the following steps:

[0106] S2111: extracting radical information of the Chinese character sentence to determine a target radical;

[0107] S2112: Extract the target keyword of the target radical from the discrimination dictionary, then select the corresponding keyword from the Chinese character sentence based on the target keyword to form a keyword matrix, and then map each keyword in the keyword matrix to a low-dimensional real-valued vector and obtain the embedded vector of the keyword through the embedding layer to obtain the vector representation of the keyword matrix.

[0108] The radical information of a Chinese character sentence is obtained through the following formula:

[0109]

[0110] wr i = Radical_Query(w i );

[0111] Where: dict(S n ) represents the Chinese sentence S n Radical information of ;w i Represents a Chinese sentence S n Chinese characters in wr i Represents Chinese character w i Radical_Query represents a pre-built radical dictionary.

[0112] The radical dictionary is represented by the following formula:

[0113] dict(R) = {R: set(R)};

[0114] Where: dict(R) represents the radical set dictionary of radical R; set(R) represents the set of Chinese characters consisting of radical R in the training data set S.

[0115] The keyword matrix is ​​calculated using the following formula:

[0116]

[0117] Where: D represents the keyword matrix; L R represents the discrimination dictionary; set(R) represents the set of Chinese characters consisting of radical R in the training data set S;

[0118] The vector representation of the keyword matrix is ​​expressed by the following formula:

[0119] ED = get_embed(D);

[0120] Where: E Drepresents the keyword table; D represents the keyword matrix; get_embed(·) represents the embedding operation.

[0121] The classification representation and text representation are extracted by the following formula:

[0122] C s ,E s =BERT(Sentence);

[0123] Where: C s Indicates classification representation; E s Indicates text representation; Sentence indicates a Chinese sentence;

[0124] The context representation is calculated by the following formula:

[0125] Y i ={y1,y2,...,y m};

[0126]

[0127]

[0128]

[0129] Where: Y i represents the context representation of the i-th Chinese sentence; yi represents the i-th character in the Chinese sentence; s i It represents the previous legacy information obtained by the i-th Chinese character sentence through the input gate and forget gate of LSTM; Indicates forward calculation; Indicates backward calculation; Indicates that the i-1th Chinese character of the Chinese sentence is combined with the corresponding previous remaining information s i The information of the i-th Chinese character is obtained by backward calculation.

[0130] The attention weight is calculated by the following formula:

[0131]

[0132] Where: α i Represents a Chinese sentence S n The classification representation C s Or the context indicates Y s The attention weight of the i-th Chinese character vector in queryi; queryi represents the i-th Chinese character vector in query, and query is the Chinese character sentence S n The classification representation C s Or the context indicates Y s (When calculating the classification attention representation, the query is the classification representation Cs , when calculating the contextual attention representation, the query is the context representation Y s );e r Represents the keyword corresponding to the i-th Chinese character vector in the keyword matrix;

[0133] The classification attention representation and context attention representation are calculated by the following formula:

[0134]

[0135]

[0136] Where: C s ′、Y s ′ respectively represent Chinese character sentences S n The classification attention representation and contextual attention representation of α i Represents a Chinese sentence S n The classification representation C s Or the context indicates Y s The attention weight of the i-th Chinese character vector in e; r Represents the keyword corresponding to the i-th Chinese character vector in the keyword matrix.

[0137] In order to better illustrate the advantages of the technical solution of the present invention, the following experiments are disclosed in this embodiment.

[0138] 1. Dataset

[0139] CNT (from Yujun Zhou, Bo Xu, Jiaming Xu, et al. Compositional Recurrent Neural Networks for Chinese Short Text Classification): The Chinese news headline dataset contains 47,952 headlines of 32 categories (i.e. news topics) for training and 15,986 headlines for testing. After removing those headlines containing special characters that cannot be processed, we retain 47,850 training headlines and 15,950 test headlines.

[0140] THUCNews (from Li J, Sun M, Xian Z. A comparison and semi - quantitative analysis of words and character - bigrams as features in Chinese text categorization): Filtered and generated based on the historical data of Sina News RSS subscription channels from 2005 to 2011, containing 740,000 news documents (2.19GB). We extracted 200,000 news titles from THUCNews, with the text length between 20 and 30. There are a total of 10 categories, 20,000 in each category. The dataset is divided into 180,000 for training, 10,000 for validation, and 10,000 for testing.

[0141] FCT: The source is the Natural Language Processing Group of the International Database Center, Department of Computer Information and Technology, Fudan University. The test corpus has a total of 9,833 documents, and the training corpus has a total of 9,804 documents, divided into 20 categories. The training corpus and the test corpus are basically divided in a ratio of 1:1.

[0142] 2. Data preprocessing

[0143] Data processing cleans and standardizes Chinese character sentence data. This process will make our data noise - free and can be directly analyzed. First, confusable characters such as spaces and line breaks are cleaned. Then, the function of simplified - to - traditional and traditional - to - simplified conversion is implemented. Here, we refer to the method proposed by Pang Zhenjun based on the comparison table and semantic relevance, and construct a comparison table and a word - segmentation module. In terms of the radical dictionary, in order to ensure the reliability of the content, we use Xinhua Dictionary to generate radical mappings.

[0144] The corresponding relationship between traditional and simplified Chinese characters in the simplified - to - traditional and traditional - to - simplified comparison table mainly refers to the "Chinese Character Simplification - Traditionalization Comparison Manual" made by Xu Hao. For the situation of "one simplified character corresponding to multiple traditional characters", it is a more effective method to solve it from the word level. We introduce a word - segmentation module in the system to match at the word - level standard. To improve the matching efficiency, a binary - tree method is adopted to judge by characters; at the same time, the longest phrase length of each one - to - many simplified Chinese character is saved to more precisely control the number of comparisons, reduce the time complexity, and improve the efficiency during matching. Taking the character "干" as an example, the constructed tree part is as Figure 4 .

[0145] In terms of semantic relevance, "Revised National Chinese Dictionary" is referred to, and at the same time, weights are set according to the influence of different words on the converted characters. For example, when "一出" appears, but if book titles "《》" appear around it, it is more inclined to be translated as "一齣". In addition, according to the spatio - temporal relationship, in order to avoid interference from redundant information, the influence range of the context on the converted characters is set to 10.

[0146] 3. Comparison Model

[0147] TextCNN (from Kim Y. Convolutional Neural Networks for Sentence Classification): TextCNN learns text by cross-spatial recognition patterns. Because of its local receptive field characteristics, it works well when it is important to detect local and position-invariant patterns. TextCNN has only one layer of convolution, which uses only one convolution on the word vectors obtained from the unsupervised neural language model (i.e. word2vec), one layer of max-pooling, and finally connects the output to softmax for classification.

[0148] TextRNN (from Toma′sˇMikolov, Kombrink S, Luka′sˇBurget, et al. EXTENSIONSOF RECURRENT NEURAL NETWORK LANGUAGE MODEL): TextRNN is a common recurrent neural network that processes tags sequentially. After training, it can have the effect of cross-time recognition and works well in NLP tasks that require understanding of remote semantics (such as part-of-speech tagging or QA).

[0149] TextRCNN (from Lai S, Xu L, Liu K, et al. Recurrent convolutional neural networks for text classification): TextRCNN first uses bidirectional LSTM to obtain the semantic and grammatical information of the input text, then uses maximum pooling to automatically filter out the most important features, and finally connects the fully connected layer to achieve classification.

[0150] FastText

[30] : FastText uses bag-of-words, n-gram features, and subwords to represent text information, and shares information between categories through hidden representations. In text classification tasks, fastText (shallow networks) can often achieve accuracy comparable to that of deep networks, but the training time is many orders of magnitude faster than that of deep networks.

[0151] BERT: A landmark pre-training model that provides important ideas for later models. BERT is usually applied to English data sets with good results. We fine-tune it and compare it with Chinese corpus.

[0152] RAFG: RAFG is a multi-granularity (RAFG) model based on radical awareness and attention mechanism, which combines four feature granularities of Chinese characters, words, character-level radicals and word-level radicals to realize Chinese sentence classification.

[0153] RAM: RAM is a new associative model guided by radicals. It takes into account the characteristics of characters, human cognitive process of characters and prior concepts with the help of language dictionaries. It simulates the real thinking process of people when they understand Chinese characters and sentences.

[0154] 4. Experimental results

[0155] The experimental results of the classification model proposed in the present invention (hereinafter referred to as CISM) on three data sets are shown in Table 1. From this, we can see that our model can achieve the best results on the three data sets in terms of accuracy, recall rate and F1 score. This proves that CISM has a better understanding of Chinese character sentences, thereby improving performance.

[0156] First, for Chinese data sets, BERT can still maintain relatively stable performance after large-scale pre-training, which proves that character-level features can also play a good role after large-scale pre-training. Secondly, TextRCNN has excellent understanding of context, and its performance is stronger than that of general neural networks. For general tasks, FastText has an extremely fast training speed while ensuring the benchmark accuracy, so when the accuracy requirements are not high (the business volume is not large), FastText can be used as the basis for implementation.

[0157] Accuracy represents the ratio of the number of samples correctly classified by the classifier to the total number of samples for a given test data set.

[0158] For a classification model f and a test set D of size n, Accuracy is defined as:

[0159]

[0160] The recall rate, from the perspective of actual results, indicates how much correct data in the test set has been selected.

[0161] Table 1 compares the accuracy of each model on the dataset

[0162]

[0163] The recall rates of each model on the dataset are shown in Table 2.

[0164] The experimental results show that compared with the RAFG model, which also uses radicals to improve text classification accuracy, CISM has comprehensively improved the performance of Chinese character sentence classification. Compared with the further improved RAM model, our CISM has also improved in accuracy, especially on the FCT dataset. It also has competitive performance in recall. This shows that the semantic enhancement algorithm that integrates the Chinese character structure has a certain effect and can better grasp the main idea of ​​Chinese character sentences. It also proves that it is more reasonable to integrate Chinese character structures such as simplified and traditional characters, radicals, etc. into the deep learning model, which can better utilize the information carried by the Chinese characters themselves, further enhance the text semantics, and thus achieve better results.

[0165] Table 2 compares the recall rates of various models on the dataset

[0166]

[0167]

[0168] In order to verify the design of CISM and determine the impact of each module on the final result, we removed each module and conducted ablation experiments on three datasets to verify the classification accuracy of two independent branches and without using auxiliary units. It can be clearly seen from Table 3 that the contribution of the attention matrix, that is, the auxiliary unit, to the overall structure is positive. Without using the attention mechanism, the accuracy of the model on the three datasets has dropped significantly. For the two independent branches, the classification of sentences using the converted traditional Chinese characters is generally better than using the initial simplified Chinese characters, and the experimental results of traditional Chinese characters are also very close to the final CISM results, which proves the practicality of the simplified-traditional conversion process for semantic enhancement. At the same time, it may also be due to problems such as parameter adjustment, data preprocessing, or semantic loss during the conversion process during training, which leads to the final result of the dual-branch structure being better than the traditional structure alone.

[0169] Table 3 Ablation experiment results

[0170]

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. A Chinese character sentence classification method based on semantic information enhancement, characterized in that: include: S1: Obtain the Chinese character sentence to be classified; S2: Input the Chinese character sentence to be classified into the trained classification model and output the corresponding predicted classification label; When training the classification model, first, a discrimination dictionary associated with radicals is constructed based on the training data set; secondly, simplified and traditional Chinese characters are converted into Chinese characters in the training data set to obtain simplified and traditional Chinese character sentences; then, the classification representation and contextual representation of simplified and traditional Chinese character sentences are extracted; at the same time, the discrimination dictionary is combined to determine the keywords in the Chinese character sentences, and the attention mechanism is combined to assign attention weights to each keyword, and then the classification representation and contextual representation are combined to calculate the final representation of the simplified and traditional texts; finally, the predicted classification labels are generated based on the final representations of the simplified and traditional texts, respectively, and then the predicted classification labels of the simplified and traditional texts are combined to obtain the overall predicted classification labels, and the model parameters of the classification model are optimized based on the overall predicted classification labels; In step S2, a discrimination dictionary is constructed by the following steps: S201: For each Chinese character sentence in the training data set: first, obtain the number of occurrences of each radical in the corresponding Chinese character sentence; then select radicals in descending order of the number of occurrences, and calculate the cosine similarity between the Chinese character corresponding to each radical and the corresponding Chinese character sentence; finally, generate a Chinese character similarity set for each radical; S202: merging the Chinese character similarity sets corresponding to the radicals in each Chinese character sentence to form a similarity matrix corresponding to the radicals; then calculating the variance of all cosine similarities of the corresponding Chinese characters in the similarity matrix corresponding to the radicals as the discrimination of the Chinese character; S203: for each Chinese character sentence: arranging the discrimination degrees of each Chinese character corresponding to each radical in descending order to obtain a discrimination degree descending list of each radical, and then integrating the discrimination degree descending lists of each radical to obtain a discrimination degree descending list of the Chinese character sentence; S204: Accumulate the descending list of discrimination of each Chinese character sentence, and arrange the discrimination of each Chinese character corresponding to each radical in descending order to obtain a discrimination dictionary of each radical; In step S204, the discrimination dictionary is expressed by the following formula: L Rni,1 =C Rni,1 ; Where: L R represents the discrimination dictionary of radical R; n represents the number of Chinese character sentences in the training data set; L Ri represents the descending list of the discrimination of the radical R in the i-th Chinese character sentence. The descending list of discrimination is an array with two columns. The first column is the Chinese character and the second column is the discrimination of the corresponding Chinese character. Rni,1 Indicates the first column of the ith row in the descending order of discrimination, L Rni,2 Represents the second column of the ith row of the descending list of discrimination; C Rni represents the cosine similarity between the ith Chinese character in the nth Chinese character sentence and the Chinese character sentence; C Rni,j represents the cosine similarity between the ith Chinese character in the nth Chinese character sentence and the jth Chinese character in the Chinese character set set(R); m represents the number of Chinese characters in the Chinese character set set(R); The final text representation is generated through the following steps: S211: determining keywords in the Chinese character sentence based on the discrimination dictionary and generating a keyword matrix; S212: inputting the Chinese sentence into the pre-trained BERT model, outputting the classification representation and text representation of the Chinese sentence; then inputting the text representation into the BiLSTM model to generate the contextual representation of the Chinese sentence; In step S212, the classification representation and text representation are extracted by the following formula: C s ,E s =BERT(Sentence); Where: C s Indicates classification representation; E s Indicates text representation; Sentence indicates a Chinese sentence; The context representation is calculated by the following formula: Y i ={y1,y2,...,y m }; Where: Y i Represents the context representation of the i-th Chinese character sentence; y i Represents the i-th character in a Chinese sentence; s i It represents the previous legacy information obtained by the i-th Chinese character sentence through the input gate and forget gate of LSTM; Indicates forward calculation; Indicates backward calculation; Indicates that the i-1th Chinese character of the Chinese sentence is combined with the corresponding previous remaining information s i The information of the i-th Chinese character obtained by backward calculation; S213: Using the keyword matrix as the key and value of the attention module, and then using the classification representation and context representation of the Chinese character sentence as the query of the attention module respectively; S214: Match each Chinese character vector in the query to the keyword in the keyword matrix one by one; then calculate the attention weight of each keyword, and then combine the classification representation and the context representation to calculate the corresponding classification attention representation and context attention representation respectively; S215: concatenating the classification attention representation, the context attention representation, the classification representation, and the context representation of the Chinese character sentence to obtain a final text representation of the Chinese character sentence; S3: Generate the classification result of the Chinese character sentence to be classified based on the predicted classification label output by the classification model.

2. The Chinese character sentence classification method based on semantic information enhancement as claimed in claim 1, characterized in that: In step S201, the similarity matrix is ​​calculated by the following formula: C Rn =cos_sim(set R (S n )×set(R)); Where: C Rn Represents the similarity matrix of radical R; cos_sim represents the calculation of cosine similarity; set R (S n ) represents the Chinese sentence S n set(R) represents the set of Chinese characters consisting of radical R in the training data set S.

3. The Chinese character sentence classification method based on semantic information enhancement as claimed in claim 2, characterized in that: In step S211, a keyword matrix is ​​generated through the following steps: S2111: extracting radical information of the Chinese character sentence to determine a target radical; S2112: extracting a target keyword of a target radical from a discrimination dictionary, and then selecting corresponding keywords from the Chinese character sentence based on the target keyword to form a keyword matrix.

4. The Chinese character sentence classification method based on semantic information enhancement as claimed in claim 3 is characterized in that: In step S2111, the radical information of the Chinese character sentence is obtained by the following formula: wr i =Radical_Query(w i ); Where: dict(S n ) represents the Chinese sentence S n Radical information of ;w i Represents a Chinese sentence S n Chinese characters in wr i Represents Chinese character w i Radical_Query represents a pre-built radical dictionary.

5. The Chinese character sentence classification method based on semantic information enhancement as claimed in claim 3 is characterized in that: In step S2112, the keyword matrix is ​​calculated by the following formula: Where: D represents the keyword matrix; L R represents the discrimination dictionary; set(R) represents the set of Chinese characters consisting of radical R in the training data set S.

6. The Chinese character sentence classification method based on semantic information enhancement as claimed in claim 1, characterized in that: In step S214, the attention weight is calculated by the following formula: Where: α i Represents a Chinese sentence S n The classification representation C s Or the context indicates Y s The attention weight of the i-th Chinese character vector in ; query i Represents the i-th Chinese character vector in the query, where query is the Chinese character sentence S n The classification representation C s Or the context indicates Y s ; e r Represents the keyword corresponding to the i-th Chinese character vector in the keyword matrix; The classification attention representation and context attention representation are calculated by the following formula: Where: C s ′、Y s ′ respectively represent Chinese character sentences S n The classification attention representation and context attention representation of αi represent the Chinese character sentence S n The classification representation C s Or the context indicates Y s The attention weight of the i-th Chinese character vector in e; r Represents the keyword corresponding to the i-th Chinese character vector in the keyword matrix.

Images