Chinese abstract innovation point classification and identification method and system based on ERNIE-BiGRU-Attention

Abstract

The invention provides a Chinese abstract innovation point classification and identification method and system based on ERNIE-BiGRU-Attention, and belongs to the technical field of natural language processing. Comprising the following steps: collecting Chinese abstract statements, and preprocessing the collected Chinese abstract statements; the preprocessed Chinese abstract statement is input into the trained ERNIE-BiGRU-Attention model to be processed, and weighted feature representation is obtained; inputting the weighted feature representation into a softmax layer for innovation point classification to obtain a classification result; the method comprises the following steps: embedding an input Chinese abstract statement into a high-dimensional space through an ERNIE pre-training model, and capturing a semantic relationship between vocabularies; the BiGRU has a bidirectional structure and can consider contextual information before and after a text sequence at the same time, which means that in each time step, the model can acquire information forward and backward from the current time step respectively, so that semantics and structures in the text can be understood more comprehensively.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and specifically relates to a method and system for classifying and identifying innovative points in Chinese abstracts based on ERNIE-BiGRU-Attention. Background Art

[0002] In recent years, with the rapid growth of the number of academic papers, how to effectively find the content you need from a large number of academic papers has become an urgent problem to be solved. As a summary of the content of academic papers, abstracts play an indispensable role in the process of finding related research, so the classification of innovation points in abstracts has also become a very important part of the field of natural language processing.

[0003] Early research on the classification of innovation points in paper abstracts relied on manually formulated rules, which classified and identified innovation points in paper abstracts based on keywords, grammatical structures or semantic patterns, but this simple and mechanical method obviously could not bring good classification results. With the development and progress of machine learning technology, researchers began to use algorithms such as support vector machines (SVM), decision trees, and random forests to improve the accuracy of classification of innovation points in paper abstracts. The existing classification of innovation points in paper abstracts is mainly based on deep learning methods, which is due to the remarkable achievements of deep learning in the field of natural language processing. YANNLC et al. applied convolutional neural networks (CNN) to the task of classifying innovation points in Chinese abstracts, which significantly improved the accuracy of classification of innovation points in abstracts. Mikolov et al. proposed to apply recurrent neural networks (RNN) to the task of classifying innovation points in Chinese abstracts. Compared with CNN using convolutional layers to extract features, the output of RNN at the current time is jointly affected by the output at the previous time and the input at the current time, so it can better learn the context information of the word. However, as the time series grows, the RNN model is prone to the problems of gradient disappearance and gradient explosion. In order to make up for the shortcomings of RNN structure, many RNN-based variants have emerged, including long short-term memory network (LSTM) and gated recurrent unit (GRU). These variants have achieved remarkable results in natural language processing tasks and effectively solved the problems of gradient vanishing and gradient explosion faced by RNN. In recent years, the introduction of attention mechanism (Attention) has enabled the model to pay more attention to the key parts of Chinese text, thereby improving the accuracy of classification. In October 2018, Google released the BERT pre-trained language model, which is based on the Transformer architecture and has the ability of bidirectional encoding. It can simultaneously consider contextual information to better understand the language. The release of BERT has achieved unprecedented performance in many natural language processing (NLP) tasks at the time, basically refreshing the best performance of many tasks. It has wide versatility and can be used in a variety of NLP tasks, such as semantic understanding, named entity recognition, text classification, etc., and has been widely praised. The emergence of BERT caused a huge sensation in the field of natural language processing and was regarded as a major breakthrough in the field of NLP. The ERNIE model is a further development after the BERT model was proposed. ERNIE uses a large number of Chinese data sets in the pre-training stage and improves the logical process of the BERT pre-training model. Compared with BERT, the ERNIE model has obvious advantages. It enhances the semantic representation ability of the model by expanding the training corpus and introducing forum corpus. In addition, ERNIE learns entity concept knowledge to perform semantic representation of complete concepts in the real world. When processing Chinese text tasks, ERNIE shows wide applicability.

[0004] Chinese literature classification is an important research direction of text classification. The current literature classification is mainly based on information such as title, keywords, journal name, author and institution. With the massive growth of the number of papers and the extreme refinement of research directions, the previous retrieval of title, keyword, journal name, author and institution can no longer meet the needs of researchers to query and obtain information, so new retrieval methods are needed to meet the needs of researchers. Compared with the information given by using titles and keywords, it is less and it is impossible to give a refined classification, so we should start with the abstract of the paper. Scientific and technological papers are the embodiment of the current researchers' scientific research results, and the abstract is the refinement of the content of the paper, so it is the most important part of the paper. Through the abstract, researchers can intuitively understand the content and logical structure of the full text, and even if they do not read the full text, they can understand the main work content of the paper. Therefore, by refining the classification of the innovation points of the abstract, researchers can make accurate inquiries, and the classification of the innovation points of the abstract can improve the accuracy of Chinese literature classification, which is of great significance to the development of text classification. In order to achieve the above goals, this paper analyzes the content of the abstract sentence, so the content of the abstract is classified according to the innovation points. The innovation point emphasizes the unique contribution of an article, so the classification of the innovation point of the abstract is the classification and identification of the innovation of the abstract. At present, there is little research on the recognition of the classification of the innovation point of the abstract. Because the writing habits of authors are different, the abstract text will express diversity and flexibility. Some publications require authors to indicate subheadings before several sentences in the abstract. These subheadings indicate the functions of several sentences in the abstract. Therefore, this type of abstract is called a structured abstract; while some publications do not have such a requirement to indicate subheadings, and directly present the abstract in a paragraph of text. This type of abstract is called an unstructured abstract. At present, the recognition algorithms for realizing the classification of abstract innovation are mostly based on structured abstracts, and many abstracts are presented in an unstructured form. If the current algorithm is directly used for the classification of unstructured abstract innovation, there will still be semantic ambiguity in the structure of the abstract sentence. The ambiguity is easy to cause semantic confusion, which will make it difficult for the algorithm to fully learn the semantic information in the abstract sentence, and the final recognition accuracy will be low. Summary of the invention

[0005] The purpose of the present invention is to overcome the problem that when existing algorithms process unstructured summary sentences, the recognition accuracy is low due to the inability to capture the bidirectional context information of the text, and to provide a Chinese summary innovation point classification and recognition method and system based on ERNIE-BiGRU-Attention.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: A method for classifying and identifying innovative points of Chinese abstracts based on ERNIE-BiGRU-Attention includes the following steps: Collecting Chinese abstract sentences and preprocessing the collected Chinese abstract sentences; The preprocessed Chinese summary sentences are input into the trained ERNIE-BiGRU-Attention model for processing to obtain weighted feature representation; The weighted feature representation is input into the softmax layer for innovation point classification to obtain the classification result; The ERNIE-BiGRU-Attention model is trained in the following way: Build the ERNIE pre-training model and train it based on the pre-processed Chinese summary sentences to obtain the word vector representation of the Chinese summary sentences; The word vector representation of the Chinese summary sentence is input into the BiGRU layer for feature extraction to obtain a feature vector containing the context information of each word vector; The feature vector containing the context information of each word vector is input into the Attention layer for weighted processing to obtain the weighted feature representation; The ERNIE pre-trained model, BiGRU layer and Attention layer are combined to obtain the ERNIE-BiGRU-Attention model.

[0007] The preprocessing of the collected Chinese abstract sentences includes data cleaning and structural processing.

[0008] The ERNIE pre-training model consists of a transformer encoder and knowledge integration; the transformer encoder is used to generate word vector representation; the knowledge integration integrates the knowledge at the phrase and entity levels into the language representation through a multi-stage knowledge masking strategy to obtain the word vector representation of the Chinese summary sentence.

[0009] The transformer encoder adopts a full attention mechanism.

[0010] The full attention mechanism formula of the Transformer encoder is:

[0011] Among them, Q, K, and V represent the input word vector matrix, and the softmax function is used to normalize the weights. represents the dimensions of query and key, and T represents transpose.

[0012] The BiGRU layer includes a forward GRU and a reverse GRU. The forward GRU is used to capture information before the current word, and the reverse GRU is used to capture information after the current word. The combination of the two can obtain complete context information.

[0013] The GRU improves the flow of information and management of memory by updating gates and resetting gates.

[0014] The calculation formula of the BiGRU layer is as follows: The hidden state ht1 of the BiGRU layer at time t is determined by the input at time t , the positive hidden state at time t-1 and the reverse hidden state output The three parts jointly determine the weighted sum of the GRU hidden states in two directions. The formula is as follows:

[0015]

[0016]

[0017] in, is the output state of the forward GRU at time t, is the output state of the reverse GRU at time t. The GRU() function represents the nonlinear transformation of the input word vector into the corresponding GRU hidden state. , is the weight matrix corresponding to the forward hidden state and the backward hidden state, is the bias vector corresponding to the hidden state at time t.

[0018] The formula for weighted processing in the Attention layer is:

[0019]

[0020]

[0021] Where: express and The correlation or match between Indicates the first elements, W and V represent learnable weight matrices, and b is the bias term; is the attention weight, indicating the The importance of each element to the query vector q, N represents the total number of items, and exp represents the exponential function.

[0022] A Chinese abstract innovation classification and recognition system based on ERNIE-BiGRU-Attention, including: A collection and preprocessing module, used for collecting Chinese abstract sentences and preprocessing the collected Chinese abstract sentences; The model training module is used to input the preprocessed Chinese summary sentences into the trained ERNIE-BiGRU-Attention model for processing to obtain weighted feature representation; The innovation point classification module is used to input the weighted feature representation into the softmax layer to classify the innovation points and obtain the classification results.

[0023] Compared with the prior art, the present invention has the following beneficial effects: The invention provides a method and system for classifying and identifying innovative points in Chinese abstracts based on ERNIE-BiGRU-Attention, comprising the following steps: collecting Chinese abstract sentences, and preprocessing the collected Chinese abstract sentences; inputting the preprocessed Chinese abstract sentences into a trained ERNIE-BiGRU-Attention model for processing to obtain weighted feature representations; inputting the weighted feature representations into a softmax layer for innovative point classification to obtain classification results; wherein the ERNIE-BiGRU-Attention model is trained in the following manner: building an ERNIE pre-training model, and training the ERNIE pre-training model based on the preprocessed Chinese abstract sentences to obtain word vector representations of the Chinese abstract sentences; inputting the word vector representations of the Chinese abstract sentences into a BiGRU layer for feature extraction to obtain a feature vector containing context information of each word vector; inputting the feature vector containing context information of each word vector into an Attention layer for weighted processing to obtain a weighted feature representation; and combining the ERNIE pre-training model, the BiGRU layer and the Attention layer to obtain the ERNIE-BiGRU-Attention model. The input Chinese summary sentence is embedded into a high-dimensional space through the ERNIE pre-training model, and the semantic relationship between words is captured; BiGRU has a bidirectional structure and can simultaneously take into account the contextual information before and after the text sequence, which means that at each time step, the model can obtain information from the current time step forward and backward, respectively, so as to more comprehensively understand the semantics and structure in the text; combined with the Attention mechanism, the model can focus more on the important parts of the text, improve the ability to grasp and understand the semantics of the text, and effectively improve the accuracy and efficiency of the classification of innovative points in the Chinese summary. The advantage of this model lies in the integration of rich information, allowing it to better adapt to the context of the Chinese summary field, thereby improving the recognition rate.

[0024] Furthermore, by introducing the Attention mechanism, the model can be helped to better process long texts, enabling the model to better capture key information when processing long sequences and avoid information loss or performance degradation.

[0025] Furthermore, ERNIE-BiGRU can capture richer semantic information, which helps to improve the performance of the model in the task of classifying innovative points in Chinese summaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of the present invention; Figure 2 This is the structure diagram of the ERNIE-BiGRU-Attention model of the present invention; Figure 3 It is the structural diagram of the ERNIE model in the present invention; Figure 4 A schematic diagram of the three stages of knowledge integration in the present invention; Figure 5 This is the structural diagram of the GRU model of the present invention; Figure 6 This is the BiGRU model structure diagram of the present invention; Figure 7 This is the structural diagram of the Attention mechanism of the present invention; Table 1 shows the experimental environment configuration of Example 3; Table 2 shows the experimental comparison results using different models. DETAILED DESCRIPTION

[0027] In order to further understand the content of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.

[0028] like Figure 1~2 As shown in FIG. 1 , a method for classifying and identifying innovative points of Chinese abstracts based on ERNIE-BiGRU-Attention includes the following steps: S1: Collect Chinese abstract sentences and preprocess the collected Chinese abstract sentences; S2: Input the preprocessed Chinese summary sentence into the trained ERNIE-BiGRU-Attention model for processing to obtain the weighted feature representation; S3: Input the weighted feature representation into the softmax layer for classification to obtain the classification result; The ERNIE-BiGRU-Attention model is classified by the following steps: S21: Build the ERNIE pre-training model and train it based on the pre-processed Chinese summary sentences to obtain the word vector representation of the Chinese summary sentences; S22: Input the word vector representation of the Chinese summary sentence into the BiGRU layer for feature extraction to obtain a feature vector containing the context information of each word vector; S23: Input the feature vector containing the context information of each word vector into the Attention layer for weighted processing to obtain a weighted feature representation; S24: Combine the ERNIE pre-trained model, BiGRU layer and Attention layer to obtain the ERNIE-BiGRU-Attention model.

[0029] Specifically, in S1, the collected Chinese abstract sentences are preprocessed, including data cleaning, standardizing and structuring these Chinese abstract sentences, and cleaning format symbols such as "\n", "\t", and spaces in the abstracts to retain the complete text content of the original data.

[0030] Specifically, in S21, ERNIE (Enhanced Representation through knowledge Integration) is a pre-trained language representation model developed by Baidu. It is an improvement and enhancement based on the BERT (Bidirectional Encoder Representations from Transformers) model developed by Google. ERNIE introduces knowledge graph information on the basis of BERT, and further improves the model's language representation ability through knowledge enhancement. In addition, ERNIE introduces entity, phrase, and word-level knowledge during the pre-training process, enabling the model to better understand and generate language.

[0031] The structure of the ERNIE model is mainly divided into two parts: transformer encoding and knowledge integration. The former uses transformer as the basic encoder to generate word vector representation to retain the contextual information of the word in the text; the latter integrates the knowledge at the phrase and entity levels into the language representation through a multi-stage knowledge masking strategy.

[0032] like Figure 3 As shown in Figure 2, the output of the ERNIE model contains the word vector representation of the text context information, and each word vector [T1, T2, T3, …, T n] contains the text information of the entire sequence. Traditional language model training is mainly based on predicting the next word, while the use of bidirectional encoding technology will cause the word to be predicted to indirectly "perceive itself" between multiple layers of context. In other words, if you want to predict the input E at time t, n , then each input will see the target E in the hidden layer n information, which will cause information leakage. To solve this problem, the BERT model adopts the strategy of replacing words at certain positions in the input sequence with [mask] tags to randomly mask part of the input sequence. On this basis, the ERNIE model made further improvements and introduced a multi-stage knowledge masking strategy to expand the scope of masking to the phrase and entity levels.

[0033] 1) Transformer encoder: The ERNIE model is a language model built on a multi-layer bidirectional Transformer encoder. The Transformer encoder uses a full attention mechanism. The attention mechanism is similar to the principle of human understanding of sentences. It can improve the speed of training the model and understand the overall meaning by focusing on the key points in the sentence. Its principle is shown in formula (1):

[0034] Where Q, K, and V represent the matrix representation of query, key, and value respectively, and the softmax function is used to normalize the weights. represents the dimensions of query and key, and T represents transpose.

[0035] In the entire calculation process, the Transformer encoder establishes a direct connection between any two words in the sentence through a series of calculation steps, and performs a weighted summation of the representations of all words. This weight is obtained by calculating the dot product of the representation of the word and the representation of other words through the softmax layer, which significantly shortens the distance between long-distance dependent features, thereby effectively improving the utilization of features.

[0036] 2) Knowledge Integration The ERNIE model further enhances the semantic representation of complete concepts in sentences by learning entity concept knowledge. To achieve this function, the model adopts a multi-stage knowledge masking strategy. like Figure 4As shown in the figure, in the first stage of the ERNIE model, the basic-level masking stage, the entire sentence is regarded as a sequence of basic language units and masked by characters and marked as [mask]. However, this method only performs random masking at the character level, so it is difficult to fully model high-level semantic information. Because character-level masking may not capture semantic relationships above the word level, such as the semantics of phrases, phrases, or entities. Therefore, masking only at the character level may limit the model's ability to understand and model the semantics of sentences.

[0037] The second stage of the ERNIE model, the entity-level masking stage, includes processing entity information that can be represented by appropriate names, such as people, places, organizations, and products. In this stage, the named entity components are first analyzed, and then the entities in the sentence are randomly selected, and each word in the entity is marked with a [mask] and then predicted. This process helps the model better understand and process named entities and improves the modeling ability of entity-level semantics. By masking and predicting named entities, the model can have a deeper understanding of the specific entity information in the sentence, thereby improving the grasp of the sentence semantics.

[0038] In the third stage of the ERNIE model, the phrase-level masking stage, a phrase is used as the basic masking unit. In this stage, the model randomly selects a phrase in the sentence, marks all the words in the phrase as [mask], and then predicts it. This processing method helps to retain the semantic information at the phrase level and better preserve the phrase information during the word embedding process. By masking and predicting at the phrase level, the model can better capture the semantic relationships above the phrase level, thereby improving the ability to understand the semantics of the sentence.

[0039] After the above three stages of processing, the ERNIE model can obtain a word vector representation composed of rich semantic information. This processing method not only helps to preserve the correlation between the various components in the sentence, but also ensures that the semantic information of its important components will not be lost while masking the complete important information. Through multiple levels of masking and prediction, the ERNIE model can better capture the semantic information of the sentence, thereby improving the understanding and modeling capabilities of the sentence semantics. This method of comprehensively utilizing semantic information at different levels enables the model to understand the text more comprehensively and provide more valuable semantic representations for downstream tasks.

[0040] Specifically, in S22, the Recurrent Neural Network (RNN) performs well in processing sequence data such as text. Its variant structure LSTM (Long Short Term Memory, LSTM) solves the RNN gradient vanishing and gradient exploding problems. However, the calculation of LSTM is relatively complex, and the training time will increase as the amount of data increases. The Gate Recurrent Unit (GRU) is a variant of LSTM. It simplifies the structure of LSTM, merges the input gate and forget gate in LSTM into a single update gate, and also mixes the cell state and hidden layer state together, solving problems such as gradient explosion in long-term memory and backpropagation. The gating mechanism of GRU mainly relies on the update gate and reset gate. The update gate of GRU filters the previous data according to the current data, and the reset gate processes the previous information as new input. Compared with the gating mechanism of the input gate, forget gate, and output gate of LSTM, the structure of GRU is simpler and requires fewer parameters. While reducing the computational complexity and increasing the model training speed, it also reduces the possibility of overfitting. The model structure of GRU is as follows: Figure 5 shown.

[0041] The workflow of the GRU model is as follows: First, the sequence data passes through the input layer, where each time step or word is converted into a vector representation and input into the model. Next, the bidirectional recurrent layer (BiRNN) is used to process forward and reverse information simultaneously to capture the previous and next relationships in the sequence. The gated recurrent unit (GRU) is used to control the flow of information and the update of memory, especially when dealing with long-term dependencies. During the forward propagation process, the generated hidden state vector contains the current input and previous context information. Finally, the merged hidden state is passed through the output layer for classification prediction.

[0042] The update gate calculation steps are as follows:

[0043] represents the gate value of the update gate, and and is the parameter related to the update gate. In GRU, the hidden state of the previous time step Will be with parameters Multiply, and the input vector of the current time step Will be with parameters Multiply, and then the two results are added and fed into the activation function to get the gate value of the update gate. Through this process, GRU can control how the input information of the current time step and the hidden state information of the previous time step are integrated. The update gate determines the importance of the new information to the current state, allowing the model to selectively update or retain information at each time step to adapt to different input situations.

[0044] The reset gate calculation steps are as follows:

[0045] Resets the gate value, while and is a parameter related to the reset gate. In GRU, the hidden state of the previous time step will be compared with the input vector of the current time step Concatenate and then with the weight parameter Multiply, plus the bias parameter The result is fed into the activation function to obtain the gate value of the reset gate. Through this process, GRU can control the extent to which the hidden state information of the previous time step is forgotten in the current time step. The role of the reset gate is to allow the model to selectively forget or retain past information to adapt to the input of the current time step.

[0046] The candidate memory calculation steps are as follows:

[0047] W and U are set as the parameters of the candidate memory, and are compared with the hidden state of the previous moment. , the input vector at the current moment , the calculation of the gate value of the reset gate and the role of the activation function, the candidate memory is obtained This candidate hidden state is controlled by the reset gate and the update gate, which includes the forgetting of the previous hidden state and the integration of new information. Therefore, it can be regarded as an intermediate result for calculating the final hidden state.

[0048] The output calculation steps are as follows:

[0049] Candidate Memory By comparing the hidden state at the previous moment It is calculated by multiplying the update gate value and the reset gate value respectively, and combining these two products to get the final output The final hidden state is calculated by the candidate hidden state and the update gate, which contains the model's overall memory and information extraction of the current time step input, thereby effectively retaining the time series characteristics of the text and enabling the model to better understand and process text data.

[0050] The unidirectional GRU can only learn information before the current moment, but cannot learn information after the current moment. Therefore, the present invention adopts a bidirectional gated recurrent unit (BiGRU) to replace the traditional GRU, and uses a bidirectional information retention method to obtain the connection between a word in the context. In the text classification scenario, BiGRU can obtain the semantic features of a word in the text through the context information, and retain the information of the word in the context as much as possible, thereby improving the classification accuracy. The model structure of the BiGRU layer is as follows: Figure 6 shown.

[0051] in, is the output state of the forward GRU at time t, is the output state of the reverse GRU at time t. The hidden state ht1 of the BiGRU layer at time t is determined by the input at time t , the positive hidden state at time t-1 and the reverse hidden state output The three parts jointly determine the weighted sum of the GRU hidden states in two directions. The formula is as follows.

[0052]

[0053]

[0054]

[0055] Among them, the GRU() function represents the encoding of the input word vector into the corresponding GRU hidden state through nonlinear transformation. , is the weight matrix corresponding to the forward hidden state and the backward hidden state, is the bias vector corresponding to the hidden state at time t.

[0056] Specifically, in S23, the Attention mechanism simulates the way humans allocate attention when processing information, allowing the model to pay more attention to input information related to the current task, thereby improving the accuracy of prediction. In the BiGRU model, the attention mechanism is combined to assign different weights to hidden states at different time steps, which helps to improve the model's weighting of the importance of the input sequence. The structure of the Attention mechanism is as follows: Figure 7 shown.

[0057] In the figure: X = [X1, X2, ..., X N ] represents N input vectors, q represents the query vector, which is used to find and select certain information in X. The softmax function is used to convert the attention scores into attention weights. The specific calculation formulas are shown in (9)-(11).

[0058]

[0059]

[0060]

[0061] Where: express and The correlation or match between Indicates the first elements, W and V represent learnable weight matrices, and b is the bias term; is the attention weight, indicating the The importance of each element to the query vector q, N represents the total number of items, and exp represents the exponential function.

[0062] Embodiment 2: A Chinese abstract innovation classification and recognition system based on ERNIE-BiGRU-Attention, including: A collection and preprocessing module, used for collecting Chinese abstract sentences and preprocessing the collected Chinese abstract sentences; The model training module is used to input the preprocessed Chinese summary sentences into the trained ERNIE-BiGRU-Attention model for processing to obtain weighted feature representation; The innovation point classification module is used to input the weighted feature representation into the softmax layer to classify the innovation points and obtain the classification results.

[0063] Embodiment 3: This example uses a labeled Chinese abstract dataset to verify and analyze it. The experimental environment and configuration are shown in Table 1.

[0064] Table 1 Experimental environment configuration

[0065] Experimental data set selection: The experimental data set of the present invention is a collection of Chinese abstracts selected from the Journal of Electronics and the Journal of Computers, which includes the purpose, background, method, result, conclusion and other information of the abstract. The present invention divides Chinese abstracts into four categories (new methods proposed completely independently, performance improvement of existing methods, migration and application of existing methods, synthesis of new methods from existing methods). The dataset consists of 10,216 samples of various types, and the dataset is divided into training set, validation set and test set in a ratio of 8:1:1. The amount of data for each abstract type in the dataset is kept consistent to ensure a balanced distribution of types.

[0066] Experimental process: The classification task is mainly divided into two training modes: one is to adjust the data set composition, randomly extract from the four categories in proportion and extract from the total sample in proportion, and then put the scattered data into the model for testing; the other is to continuously adjust the parameters of the model to obtain the best effect parameter value of the model. Considering the experimental equipment and time cost, the present invention adopts the second training method for testing, and obtains the parameter setting value of the best model performance by continuously adjusting the model parameters.

[0067] Since there are some numbers, English and some special symbols in the dataset, the dataset should be preprocessed before the experiment, some non-Chinese information should be cleaned, and then the processed dataset should be put into the model for experiment.

[0068] In order to verify the reliability of the model, the present invention conducts comparative experimental research on the following model.

[0069] All the above models have the same hyperparameters and the epoch value is 10. Under the same conditions, multiple experiments are conducted on each model, and the highest value is taken as the experimental result of the model.

[0070] Evaluation indicators: There are many evaluation indicators in natural language processing. Among them, the most common and common evaluation criteria for the classification of innovation points in Chinese abstracts generally include precision P (Precision), recall R (Recall) and F1 (F1-Score) value. Precision is also called precision rate, which indicates the probability of actually positive samples among all samples predicted as positive, reflecting how many of the samples predicted as positive are correct. Recall is also called recall rate, which indicates the proportion of all actual positive samples that are correctly predicted as positive. The higher the recall rate, the fewer positive samples are missed.

[0071] F1 value is the harmonic mean of precision and recall, and is used to comprehensively evaluate the performance of a classifier. When the gap between precision and recall is large, F1 value can provide a balanced evaluation. The specific calculation formulas are shown in (12)-(14).

[0072]

[0073]

[0074]

[0075] Where: TP: the number of data when both the actual value and the predicted value are positive; FP: the number of data when the actual value is negative and the predicted value is positive; FN: The number of data when the actual value is positive and the predicted value is negative.

[0076] Experimental results: Based on the constructed experimental data set, the present invention uses the ERNIE-BiGRU-Attention model to conduct a Chinese abstract innovation classification experiment. In order to verify the effect of the model of the present invention, BERT, BERT-BiLSTM, BERT-DPCNN, ERNIE, and ERNIE-BiGRU are selected as experimental control models. The experimental results are shown in Table 2.

[0077] Table 2 Experimental comparison results

[0078] As shown in Table 2, by comparing the performance of different Chinese abstract innovation classification models, the model proposed by the present invention is found to be the most superior in various indicators. The precision rate is at least 0.55% higher than that of the comparison model, the recall rate is at least 0.91% higher than that of the comparison model, and the F1 value is at least 0.75% higher than that of the comparison model. This shows that the model proposed by the present invention has better performance in the task of Chinese abstract innovation classification.

[0079] The analysis of a single model shows that the BERT model has poor performance. Although BERT uses the Transformer encoder to achieve bidirectional context modeling of the input text through the self-attention mechanism, so that the model can better understand the meaning of words in different contexts, it relies on a large-scale unsupervised corpus for pre-training without explicitly incorporating external knowledge. Although the DPCNN model learns hierarchical features through a deep convolutional neural network, which helps capture local and global information in the text and improves the expressiveness of the model, when combined with the BERT model, this will make the model complex, requiring a long training time and a larger storage space. BiLSTM is a variant of a recurrent neural network (RNN) that can process sequence data and capture the dependencies between the previous and next sequences. However, when combined with BERT, its generalization ability in specific fields or specific tasks may be limited, and targeted adjustments and optimizations are required. The advantage of ERNIE over BERT is that it can integrate external knowledge, better capture the semantic information in the text, make the language representation it learns richer and more accurate, and utilize additional knowledge resources, making it more advantageous when dealing with tasks that require world knowledge. GRU (Gated Recurrent Unit) is a gated recurrent neural network structure. Compared with traditional recurrent neural networks, it can model long-term dependencies more effectively, so it helps to process long text sequences. The use of BiGRU structure can better capture the bidirectional contextual information in the text, which means that the model can take into account the information before and after the words at the same time, so as to better understand the context of the text. Combined with the ERNIE pre-trained language representation model, ERNIE-BiGRU can capture richer semantic information, which helps to improve the performance of the model in the Chinese abstract innovation point classification task. By introducing the Attention mechanism, the model can better process long texts, so that the model can better capture key information when processing long sequences, avoiding information loss or performance degradation. The combination of the Attention mechanism can make the model focus more on the important parts of the text, improving the ability to grasp and understand the semantics of the text.

[0080] Effect: The ERNIE-BiGRU-Attention model has advantages in terms of richness of semantic representation, deep understanding ability, and long text processing ability. The model has learned rich language representation through large-scale text data in the pre-trained language model of ERNIE, embedding words into a high-dimensional space and capturing the semantic relationship between words. Since traditional recurrent neural networks (RNNs) are prone to gradient vanishing or explosion problems when processing long sequences, BiGRU can better capture long-term dependencies in text sequences by introducing gating mechanisms (such as reset gates and update gates in GRU). BiGRU has a bidirectional structure that can simultaneously consider the contextual information before and after the text sequence, which means that at each time step, the model can obtain information from the current time step forward and backward respectively, so as to more comprehensively understand the semantics and structure in the text. By introducing the Attention mechanism, the model can better process long texts, so that the model can better capture key information when processing long sequences, avoiding information loss or performance degradation. Combined with the Attention mechanism, the model can focus more on the important parts of the text and improve the ability to grasp and understand the semantics of the text.

[0081] In general, the ERNIE-BiGRU-Attention model uses a variety of advanced deep learning technologies such as the ERNIE pre-trained language model, bidirectional gated recurrent unit (BiGRU), and the Attention mechanism to effectively improve the accuracy and efficiency of the classification of innovation points in Chinese abstracts. The advantage of this model lies in the integration of rich information, allowing it to better adapt to the context of the Chinese abstract field, thereby improving the recognition rate.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A Chinese abstract innovation point classification and recognition method based on ERNIE-BiGRU-Attention, characterized by: The steps include: Collecting Chinese abstract sentences and preprocessing the collected Chinese abstract sentences; The preprocessed Chinese summary sentences are input into the trained ERNIE-BiGRU-Attention model for processing to obtain weighted feature representation; The weighted feature representation is input into the softmax layer for innovation point classification to obtain the classification result; The ERNIE-BiGRU-Attention model is trained in the following way: Build the ERNIE pre-training model and train it based on the pre-processed Chinese summary sentences to obtain the word vector representation of the Chinese summary sentences; The word vector representation of the Chinese summary sentence is input into the BiGRU layer for feature extraction to obtain a feature vector containing the context information of each word vector; The feature vector containing the context information of each word vector is input into the Attention layer for weighted processing to obtain the weighted feature representation; The ERNIE pre-trained model, BiGRU layer and Attention layer are combined to obtain the ERNIE-BiGRU-Attention model.

2. According to claim 1, a Chinese abstract innovation point classification and identification method based on ERNIE-BiGRU-Attention is characterized in that: The preprocessing of the collected Chinese abstract sentences includes data cleaning and structural processing.

3. According to claim 1, a Chinese abstract innovation point classification and identification method based on ERNIE-BiGRU-Attention is characterized in that: The ERNIE pre-training model consists of a transformer encoder and knowledge integration; the transformer encoder is used to generate word vector representation; the knowledge integration integrates the knowledge at the phrase and entity levels into the language representation through a multi-stage knowledge masking strategy to obtain the word vector representation of the Chinese summary sentence.

4. According to claim 3, a Chinese abstract innovation point classification and identification method based on ERNIE-BiGRU-Attention is characterized in that: The transformer encoder adopts a full attention mechanism.

5. According to claim 4, a Chinese abstract innovation point classification and identification method based on ERNIE-BiGRU-Attention is characterized in that: The full attention mechanism formula of the Transformer encoder is: Among them, Q, K, and V represent the input word vector matrix, and the softmax function is used to normalize the weights. represents the dimensions of query and key, and T represents transpose.

6. According to claim 1, a Chinese abstract innovation point classification and identification method based on ERNIE-BiGRU-Attention is characterized in that: The BiGRU layer includes a forward GRU and a reverse GRU. The forward GRU is used to capture information before the current word, and the reverse GRU is used to capture information after the current word. The combination of the two can obtain complete context information.

7. The method for classifying and identifying innovative points of Chinese abstracts based on ERNIE-BiGRU-Attention according to claim 6 is characterized in that: The GRU improves the flow of information and management of memory by updating gates and resetting gates.

8. According to claim 6, a Chinese abstract innovation point classification and identification method based on ERNIE-BiGRU-Attention is characterized in that: The calculation formula of the BiGRU layer is as follows: The hidden state ht1 of the BiGRU layer at time t is determined by the input at time t , the positive hidden state at time t-1 and the reverse hidden state output The three parts jointly determine the weighted sum of the GRU hidden states in two directions. The formula is as follows: in, is the output state of the forward GRU at time t, is the output state of the reverse GRU at time t. The GRU() function represents the nonlinear transformation of the input word vector into the corresponding GRU hidden state. , is the weight matrix corresponding to the forward hidden state and the backward hidden state, is the bias vector corresponding to the hidden state at time t.

9. The method for classifying and identifying innovative points of Chinese abstracts based on ERNIE-BiGRU-Attention according to claim 1, characterized in that: The formula for weighted processing in the Attention layer is: Where: express and The correlation or match between Indicates the first elements, W and V represent learnable weight matrices, and b is the bias term; is the attention weight, indicating the The importance of each element to the query vector q, N represents the total number of items, and exp represents the exponential function.

10. A Chinese abstract innovation classification and recognition system based on ERNIE-BiGRU-Attention, characterized by: include: A collection and preprocessing module, used for collecting Chinese abstract sentences and preprocessing the collected Chinese abstract sentences; The model training module is used to input the preprocessed Chinese summary sentences into the trained ERNIE-BiGRU-Attention model for processing to obtain weighted feature representation; The innovation point classification module is used to input the weighted feature representation into the softmax layer to classify the innovation points and obtain the classification results.

Images