A technical term extraction method for large-scale text data

Abstract

The application discloses a technical term extraction method for large-scale text data, and comprises the following steps: step S1, first, a set of term candidate words is generated, after obtaining a candidate data set, the term candidates need to be sorted and screened, so that the final technical term is determined; step S2, technical terms in text data including news text and PDF files of scientific and technological literature are extracted, for the PDF files, text content extraction is first carried out, text content is obtained, and then technical term extraction is carried out on the text content. The technical term extraction method for large-scale text data is used for technical term extraction for large-scale text data, acquires a field research theme, constructs a scientific and technological research and development event, and realizes extraction of event arguments and trigger words through an event extraction algorithm based on deep learning, so that a scientific and technological event library is formed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of technical term extraction, and particularly to a technical term extraction method for large-scale text data. BACKGROUND

[0002] A technical term refers to a key word that can represent a technical subject in a research field, is an important argument of a scientific and technological research and development event, and is a key element that represents the development degree of a scientific and technological event. A large number of technical terms are contained in abstracts and full texts of papers, patents, and other achievements, and the technical terms represent research contents and technical subjects of the achievements and can reflect the development of a specific technology in a specific field.

[0003] Technical term extraction plays a crucial role in a technical subject in a research field.

[0004] Therefore, the present application provides a technical term extraction method for large-scale text data to improve the effect of technical term extraction. SUMMARY

[0005] The present application aims to provide a technical term extraction method for large-scale text data to solve the problems in the background art.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical solution: a technical term extraction method for large-scale text data, comprising the following steps:

[0007] Step S1: First, a set of term candidate words is generated, and after obtaining a candidate data set, the term candidates are sorted and filtered to determine the final technical terms.

[0008] Step S2: Technical terms in text data, including news texts and technical literature PDF files, are extracted. For a PDF file, text content is extracted first to obtain the text content, and then technical term extraction is performed on the text content.

[0009] Step S3: A deep learning-based method is used to extract technical terms. The word embedding representation of the entire sentence is taken as input and fed into a deep learning model. Then, a deep calculation model composed of multiple processing layers learns candidate term representations with multiple abstraction levels. Finally, the representation is classified into technical terms. Specifically, the method comprises:

[0010] The Bi-LSTM-CRF deep learning model is adopted, the term extraction is regarded as a sequence labeling problem, the word vector features, the part-of-speech features and the entity features of each word in the document are extracted as input data, after being processed through a bidirectional multilayer hidden layer, the CRF method is used to map the word into one of {B, I, O, E, S} labels, and the Viterbi of the incremental self-training algorithm is used to reduce the artificial labeling cost and improve the term extraction efficiency in view of the fact that there is no large-scale mature corpus in the field, the candidate terms and context information are combined, more types of features are represented by word embedding vectors, so that better term extraction effect is achieved, and the method is suitable for larger document sets;

[0011] Step S4: after the technical term extraction is completed, the attributes of the technical terms are measured to determine whether the technical terms can represent the technical development direction of the field;

[0012] Step S5: an algorithm is designed to extract event elements and event trigger words;

[0013] Step S6: the event element extraction module better acquires remote information through an attention mechanism, and entity recognition is taken as a sequence labeling task, a classic Bi-GRU-CRF (Bi-GRU: Bi-Gated Recurrent Unit, conditional random field) is used as a basic framework, and the model is modified through requirements;

[0014] Step S7: the overall model of event trigger word extraction is based on Bi-GRU+CRF, and the event trigger words are extracted in a sequence labeling manner;

[0015] Step S8: the scheme adopting the attention mechanism adds event element entity information into the model;

[0016] Step S9: the reasonableness correlation relationship of the science and technology fog event is inferred.

[0017] Compared with the prior art, the technical term extraction method for large-scale text data has the advantages that the technical term extraction is performed on large-scale text data, the field research theme is acquired, the science and technology R&D event is constructed, the event argument and trigger word extraction are realized through the event extraction algorithm based on deep learning, and thus the science and technology event library is formed. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is the event element extraction model diagram of the application;

[0019] Figure 2 is the GRU model diagram of the application;

[0020] Figure 3It is the GCN model thought diagram of the present application;

[0021] Figure 4 It is the structure diagram of the GCN Model VAE model of the present application; DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0023] Please refer to Figures 1-4 The present application provides a technical solution: a technical term extraction method for large-scale text data, comprising the following steps:

[0024] Step S1: First, generate a set of term candidate words, after obtaining the candidate data set, the term candidate is sorted and screened to determine the final technical term;

[0025] Step S2: Extract technical terms from text data including news text and technical literature in PDF files. For PDF files, first extract the text content to obtain the text content, and then extract the technical terms from the text content;

[0026] Step S3: Use a deep learning-based method to extract technical terms. The word embedding representation of the entire sentence is taken as input and fed into a deep learning model. Then a deep calculation model composed of multiple processing layers learns the candidate term representation with multiple abstraction levels. Finally, the representation is classified into technical terms, which specifically includes:

[0027] A Bi-LSTM-CRF deep learning model is used to treat term extraction as a sequence labeling problem. The word vector features, part-of-speech features and entity features of each word in the document are extracted as input data. After processing by a bidirectional multilayer hidden layer, the CRF method is used to map the word to one of the {B, I, O, E, S} labels. Due to the lack of large-scale mature corpus in the field, the Viterbi algorithm is used to reduce the cost of manual annotation and improve the efficiency of term extraction. The candidate terms and context information are combined to represent the word embedding vector and integrate more types of features, thereby achieving better term extraction effect and being suitable for larger document sets.

[0028] Step S4: After completing the technical term extraction, the properties of the technical terms are measured to determine whether they can represent the technical development direction of the field;

[0029] Step S5: the designed algorithm extracts event elements and event trigger words;

[0030] Step S6: the event element extraction module better acquires remote information through an attention mechanism, and entity recognition is taken as a sequence labeling task, a classic Bi-GRU-CRF (Bi-GRU: Bi-Gated Recurrent Unit, conditional random field) is used as a basic framework, and the model is modified through requirements;

[0031] Step S7: the overall model of event trigger word extraction is based on Bi-GRU+CRF, and the event trigger word is extracted in a sequence labeling manner;

[0032] Step S8: the scheme adopting the attention mechanism adds event element entity information into the model;

[0033] Step S9: the reason correlation relationship of the science and technology fog event is inferred.

[0034] The application further comprises the following: the overall model in step S7 extracts seven entities in a sequence labeling manner, for a word vector, a bert word vector is selected, and a corresponding five word library of the end part of five entities of personnel, organization, achievement, project and research field is made to generate a suffix feature; the bert word vector and the suffix feature are connected together as the input of the model; next, the data passes through a Bi-GRU model to acquire the related information of the context before and after, and the length of the entity is observed; next, the self-attention mechanism can effectively accept remote information, and then the suffix feature is connected repeatedly; finally, the CRF is used to learn the implicit relationship between the BIO labels of sequence labeling, so that the final model effect is improved.

[0035] The application further comprises the following: the input of the overall model in step S7 is composed of two parts, a bert word vector and a suffix feature; the bert word vector has been proposed for many years, and after Google, people continue to improve the bert based on the full word masking, that is, not randomly masking words, but masking the whole word; and the meaning of Chinese is in units of words, so the obtained bert word vector has higher performance.

[0036] The Bi-LSTM complex model increases the model training time, a variant of LSTM, a gated recurrent unit (GRU, Gated Recurrent Unit) is proposed, in the GRU, the forgetting gate and the input gate are combined into an update gate, and the cell state and the hidden state are combined together, so that the parameter amount is reduced, the training cost is reduced, and the model training effect is ensured;

[0037] The reset gate of the GRU mainly obtains information that needs to be newly added, and first inputs the hidden state of the previous moment and the input of the current moment, obtains through linear transformation and sigmoid activation function;

[0038] The calculation process of the update gate is similar to that of the reset gate, and the hidden state of the previous moment and the input of the current moment are input, and are obtained through linear transformation and sigmoid activation function, different from the update of the LSTM, by setting the sum of the coefficients of the hidden filling and the input of the current moment in the update to be one, the parameters used for calculation are reduced;

[0039] The GRU combines the cell state and the hidden state, reduces the number of gated units by setting the coefficient in the update process to be one, gets rid of the problem of gradient disappearance or gradient explosion caused by long-term dependence, reduces the training cost of the neural network, and directly connects the outputs of the forward GRU and the reverse GRU to obtain the Bi-GRU model.

[0040] In the step S7, the overall model extracts the event trigger word in a sequence labeling manner, for the word vector, the bert word vector is selected, the initial model input only has the bert word vector, then the context information is extracted through the Bi-GRU neural network model, and the attention mechanism is reached, in the attention mechanism, the event element entity information is used as the query, the output of the Bi-GRU is used as the key and the value, the event element entity information is used to improve the event trigger word extraction effect, finally, the linear transformation and the CRF model are used, and the final output of the model is obtained, the final output of the model is still the CRF model, so the loss function calculation method of the event trigger word extraction and the event element entity extraction is consistent.

[0041] In the step S7, the overall model extracts the event trigger word in a sequence labeling manner, for the word vector, the bert word vector is selected, the initial model input only has the bert word vector, then the context information is extracted through the Bi-GRU neural network model, and the attention mechanism is reached, in the attention mechanism, the event element entity information is used as the query, the output of the Bi-GRU is used as the key and the value, the event element entity information is used to improve the event trigger word extraction effect, finally, the linear transformation and the CRF model are used, and the final output of the model is obtained, the final output of the model is still the CRF model, so the loss function calculation method of the event trigger word extraction and the event element entity extraction is consistent.

[0042] The application further comprises: the attention mechanism in step S8, based on three inputs Q, K and V, the weight for each position is obtained through Q and K, the value in V is inquired through the weight, and finally the output is obtained, which is the basic process of the attention mechanism, the actual input of the attention mechanism mainly comprises the output of the previous Bi-GRU and the information of the event element entity, the output of the Bi-GRU is converted into two matrices K and V, the information of the event element entity is converted into a Q matrix, and the method of converting the information of the event element entity into the Q matrix is similar to the method of generating the suffix feature, that is, four positions of the event element entity are obtained first, and then corresponding vectors are generated in the same way, since the vector length of the Q and K matrices needs to be consistent in the attention mechanism, the information of the event element entity is linearly transformed again, and finally the Q matrix is obtained;

[0043] The output of the Bi-GRU is divided in the middle, and two matrices are obtained, which are used as the K and V matrices of the attention mechanism, but the output of the Bi-GRU is the direct connection of the output of the forward GRU and the reverse GRU, and the direct division will separate the forward and reverse information, and what we need is that the K and V matrices each contain sufficient forward and reverse information, so a linear transformation is needed here to mix the forward and reverse outputs, and then the K and V matrices are obtained, after obtaining the three matrices Q, K and V, the calculation of the attention mechanism is performed next, and the dot product method is used to calculate the attention.

[0044] If the method of converting the information of the event element entity into the Q matrix is checked, it can be found that most of the Q matrix before the linear transformation is 0, so the information in the Q matrix after the linear transformation is still relatively sparse, which leads to poor effect of directly using the attention mechanism as subsequent input, and the information that is relatively dense is the information derived from the word vector, so the output of the attention mechanism and the output of the Bi-GRU need to be connected together at last, the information of the event element entity attention mechanism only serves as an auxiliary to improve the final extraction effect.

[0045] The application further comprises: the essence of the event relationship inference in step S9 is link prediction in the event graph, and whether there is a correlation relationship between different events is inferred, the link prediction strategy for the science mist is used, the past event information is comprehensively considered, the most likely future event can be inferred, and the link prediction strategy based on the event network mainly predicts the correlation between existing nodes.

[0046] The further, the science and technology haze event in step S9 is composed of a triple form {h, r, t}, wherein h and t are head and tail entities respectively, link prediction is performed using a graph neural network GCNModelVAE, the GCNModelVAE is based on the VGAE idea, the VGAE is a combination of variational Bayesian and a graph neural network, two neural networks are constructed to learn the mean and variance of input data respectively, so as to simulate the distribution of data, the role of the graph convolutional neural network (GCN) is to extract features in the graph structure, the GCN is also a neural network layer, and the propagation mode between layers of the GCN is:

[0047]

[0048] Wherein is an adjacency matrix,

[0049] Please refer to Figure 4 , X is a feature matrix of nodes, A is an adjacency matrix, hidden variables Z are obtained by using posterior probability, and the adjacency matrix A is reconstructed by using the hidden variables, the encoder of the VGAE is a two-layer graph convolutional network:

[0050]

[0051] The two-layer graph convolutional network is defined as follows:

[0052]

[0053] Through the encoder, the VGAE maps the adjacency matrix to a low-dimensional feature space, and then uses the decoder to calculate the probability that an edge exists between two points to reconstruct the graph.

[0054]

[0055] Although the embodiments of the present application have been shown and described, it can be understood by those skilled in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A technical term extraction method for large-scale text data, characterized by, Comprise the following steps: Step S1: First, generate a set of term candidates, after obtaining the candidate data set, the term candidate is sorted and screened, so as to determine the final technical term; Step S2: The technical terms in the text data including news text, technical literature in PDF file are extracted, for PDF file, first text content extraction is carried out to obtain text content, and then technical term extraction is carried out on the text content; Step S3: The method based on deep learning is used for extracting technical terms, the word embedding representation of the whole sentence is taken as the input, which is fed into the deep learning model, then the deep calculation model composed of multiple processing layers learns the candidate term representation with multiple abstraction levels, and finally the representation is classified according to the term category, which specifically includes: Bi-LSTM-CRF deep learning model is used, term extraction is regarded as a sequence labeling problem, word vector features, part of speech features and entity features of each word in the document are extracted as input data, after processing by double-layer hidden layer, CRF method is used to map the word to one of {B, I, O, E, S} labels, and in view of the fact that there is no large-scale mature corpus in the field, Viterbi algorithm is used to reduce the cost of manual labeling and improve the efficiency of term extraction, the candidate term and the context information are combined, more types of features are represented by word embedding vector, so as to achieve better term extraction effect, which is suitable for larger document set; Step S4: After completing the technical term extraction, the attributes of the technical term are measured to determine whether it can represent the technical development direction of the field; Step S5: An algorithm is designed to extract event elements and event trigger words; Step S6: The event element extraction module uses attention mechanism to better obtain remote information, and entity recognition is taken as a sequence labeling task, and Bi-GRU-CRF is used as the basic framework to modify the model according to the demand; Step S7: The overall model of event trigger word extraction is based on Bi-GRU+CRF, which extracts event trigger words in the form of sequence labeling; Step S8: The attention mechanism is used to add event element entity information to the model; Step S9: The reason correlation relationship of science and technology fog event is inferred. 2.The technical term extraction method for large-scale text data according to claim 1, wherein, The overall model in step S7 extracts seven entities by sequence labeling, for word vector, bert word vector is selected, and five word libraries corresponding to the end of personnel, institution, achievement, project and research field are generated to generate suffix features, the bert word vector and the suffix features are connected together as the input of the model, then the data passes through the Bi-GRU model to obtain the related information of the context, and the length of the entity is observed, then the self-attention mechanism can effectively accept the remote information, and then the suffix features are connected repeatedly, finally the CRF is used to learn the implicit relationship between the BIO labels of sequence labeling, so as to improve the effect of the final model. 3.The technical term extraction method for large-scale text data according to claim 1, wherein, The input of the overall model in step S7 is composed of two parts, bert word vector and suffix feature; The Bi-LSTM complex model increases the model training time, and therefore a variant of LSTM, a gated recurrent unit (GRU), is proposed. In the GRU, the forget gate and the input gate are combined into an update gate, and the cell state and the hidden state are combined together, thereby reducing the number of parameters and the training cost while ensuring the model training effect. The reset gate of the GRU derives information that needs to be newly added. First, the hidden state of the previous moment and the input of the current moment are input, and then linear transformation and a sigmoid activation function are performed to obtain the reset gate. The calculation process of the update gate is similar to that of the reset gate. The hidden state of the previous moment and the input of the current moment are input, and then linear transformation and a sigmoid activation function are performed to obtain the update gate. Unlike the update of the LSTM, the coefficients of the hidden state and the input of the current moment in the update are added to 1, thereby reducing the parameters used for calculation. The GRU combines the cell state and the hidden state, reduces the number of gating units by adding the coefficients in the update process to 1, and at the same time, gets rid of the problem of gradient disappearance or gradient explosion caused by long-term dependence, thereby reducing the training cost of the neural network. The outputs of the forward GRU and the reverse GRU are directly connected to obtain the Bi-GRU model. 4.The technical term extraction method for large-scale text data according to claim 1, wherein, The overall model in step S7 extracts the event trigger word in a sequence labeling manner. For the word vector, the bert word vector is selected. The initial model input is only the bert word vector, and then the context information is extracted through the Bi-GRU neural network model. The attention mechanism is reached. In the attention mechanism, the event element entity information is used as the query, the output of the Bi-GRU is used as the key and the value, the event element entity information is used to improve the event trigger word extraction effect, and finally the linear transformation and the CRF model are used to obtain the final output of the model. The final output of the model is still the CRF model, so the loss function calculation method of the event trigger word extraction and the event element entity extraction is consistent.

5. The technical term extraction method for large-scale text data according to claim 1, wherein, The attention mechanism in step S8 is based on three inputs Q, K and V. The weight for each position is obtained through Q and K, and the value in V is queried through the weight to obtain the final output. This is the basic process of the attention mechanism. The actual input of the attention mechanism includes the output of the previous Bi-GRU and the event element entity information. The output of the Bi-GRU is converted into two matrices K and V, and the event element entity information is converted into a Q matrix. The method of converting the event element entity information into the Q matrix is similar to the method of generating the suffix feature. First, the positions of the four event element entities are obtained, and then the corresponding vectors are generated in the same way. Since the vector length of the Q and K matrices needs to be consistent in the attention mechanism, the event element entity information is linearly transformed again, and finally the Q matrix is obtained. The output of the Bi-GRU is split in the middle to obtain two matrices, which are used as the K and V matrices input to the attention mechanism. However, the output of the Bi-GRU is the concatenation of the forward GRU and reverse GRU outputs. Directly splitting the output will separate the forward and reverse information, but we need the K and V matrices to each contain sufficient forward and reverse information. Therefore, a linear transformation is used to mix the forward and reverse outputs before splitting to obtain the K and V matrices. After obtaining the Q, K, and V matrices, the attention mechanism is calculated using the common dot product method. If the method of converting the event element entity information into the Q matrix is examined, it will be found that most of the Q matrix before the linear transformation is 0. Therefore, the information in the Q matrix after the linear transformation is still relatively sparse, which leads to poor results when directly using the attention mechanism as the subsequent input. In contrast, the information that is relatively dense is derived from the word vector. Therefore, the output of the attention mechanism and the output of the Bi-GRU are connected together, and the information of the event element entity attention mechanism is only used as an auxiliary to improve the final extraction effect. 6.The technical term extraction method for large-scale text data according to claim 1, wherein, The essence of the event relationship inference in step S9 is link prediction in the event graph, which infers whether there is a relationship between different events. Using the link prediction strategy for the science fog, the past event information is considered to infer the most likely future event. The link prediction strategy based on the event network mainly predicts the relationship between existing nodes. 7.The technical term extraction method for large-scale text data according to claim 1, wherein, The science fog event in step S9 is in the form of a triple {h, r, t}, where h and t are the head and tail entities, respectively. The graph neural network GCNModelVAE is used for link prediction. The GCNModelVAE is based on the VGAE idea. The VGAE is a combination of Variational Bayesian and graph neural networks. Two neural networks are constructed to learn the mean and variance of the input data, respectively, to simulate the distribution of the data. The role of the graph convolutional neural network GCN is to extract the features in the graph structure. The GCN is also a neural network layer, and the propagation method between its layers is: wherein is an adjacency matrix,

Images