Subject text entity extraction and keyword generation method and device

Abstract

The invention discloses a subject text entity extraction and keyword generation method and device, and the method comprises the steps: obtaining a global feature matrix, a local feature matrix and a head-tail reinforced feature matrix according to a plurality of character-level and word-level splicing feature vectors of an original text sequence, performing feature fusion on the matrix by using a multi-head self-attention mechanism to obtain a fused feature matrix, and decoding the fused feature matrix to obtain an existing knowledge point set; the existing knowledge point set comprises entity knowledge points existing in the original text; according to the original text sequence, generating a word embedding vector containing semantic information of the original text by utilizing BART and SoftMask, and obtaining a generated knowledge point set based on the word embedding vector; the generated knowledge point set comprises entity knowledge points which do not exist in the original text and are associated with the original text; and screening is performed according to the word importance score, the existing knowledge point set and the generated knowledge point set to obtain a more accurate and comprehensive key knowledge point set.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and specifically relates to a subject text entity extraction and keyword generation method and device. Background Art

[0002] Knowledge acquisition is one of the core steps in constructing a subject knowledge graph. It refers to the process of automatically extracting information from multi-source heterogeneous data sources to obtain knowledge elements such as entities, attributes and their relationships, thereby forming an ontological knowledge expression. In the field of education, a large amount of data exists in an unstructured form, such as subject textbooks, test questions, etc., which contain rich information and knowledge. Automatically extracting and mining valuable entity knowledge points and relationships from these unstructured text data and storing them in a structured form plays an important role in the construction of knowledge graphs in the field of education, thereby better promoting the development of smart education.

[0003] At present, there have been extensive studies on knowledge acquisition of educational texts. Taking the textbook "Principles of Computer Organization" as an example, Zhang Huinan et al. used the more mature BiLSTM-CRF (bidirectional long short-term memory network-conditional random field) model combined with manual verification to extract knowledge points, providing knowledge acquisition for the construction of subject knowledge graphs. Taking the textbook "Linear Algebra" as an example, Yang Wenxia et al. used the "pre-training + fine-tuning" strategy to extract entities and relationships to acquire knowledge for subject knowledge graphs. However, unlike general fields, educational texts usually contain a lot of professional knowledge and information. Current research still uses the method of general field text recognition. There are problems such as polysemy and difficulty in identifying rare or unknown knowledge points in the recognition and extraction of educational texts. In addition, most of the current research on knowledge acquisition of subject knowledge graphs only stays in the recognition of entity knowledge points in text content, and there is little involvement in the mining of text content. Subject texts contain rich subject domain knowledge on the basis of various semantic information of general texts, and the knowledge is highly correlated. The implicit knowledge points in subject texts have not been fully and comprehensively extracted and utilized.

[0004] Entity extraction, also known as named entity recognition (NER) in natural language processing tasks, started late in China compared to foreign countries. In addition, Chinese has complex word formation and words are not separated by spaces. Therefore, compared with English named entity recognition, Chinese named entity recognition tasks are more challenging. However, the commonly used named entity recognition methods at home and abroad are generally divided into three types: rule-based and dictionary-based methods, traditional machine learning-based methods, and deep learning-based methods.

[0005] In the field of education, named entity recognition technology mainly extracts proper nouns from teaching resources such as textbooks and test questions, and establishes corresponding sequence labels for text resources such as textbooks and test questions based on the entity categories of these proper nouns. In the process of screening text resources according to knowledge points, proper nouns appearing in the text will be abstracted as entities, and the text resources will be associated with specific knowledge points, and then the subsequent teaching resource retrieval task will be carried out.

[0006] At present, the mainstream NER technology is based on deep learning methods. The main idea of ​​this method is to take each word in a given text as the input of the model, learn the representation of each word and the semantic information of the text through a deep neural network, and predict whether each word belongs to a certain entity category. However, the text in the field of education is highly professional and contains many professional terms. The text vectorization representation model based on general fields cannot fully express the semantic information of educational texts. Therefore, deep learning research methods that combine the characteristics of the field of education have received widespread attention.

[0007] The existing methods of text entity extraction in the field of education mainly include three aspects: the first is the entity extraction method based on rules and dictionaries, the second is the entity extraction method based on machine learning models such as support vector machines and hidden Markov, and the third is the entity extraction method based on deep learning.

[0008] The entity extraction method based on rules and dictionaries is highly dependent on rules and dictionaries. Therefore, for educational subject texts containing a large number of professional terms, the quality of the constructed rules and dictionaries will play a decisive role in the entity recognition effect; and researchers need to spend most of the entire entity recognition process manually compiling key technologies such as rules and dictionaries; in addition, the scope of manually compiled rules and dictionaries is limited, and different fields require the compilation of different rules and field dictionaries, so this method has poor generalization ability and method reusability.

[0009] Compared with the entity recognition methods in the field of education based on rules and dictionaries, the above-mentioned machine learning-based methods can automatically identify entities in texts through feature engineering. However, these methods only focus on flat-structured subject entities, which is inconsistent with the characteristics of nested entities in Chinese educational texts. In addition, researchers are required to spend more than 80% of their time and energy on feature engineering work such as data preprocessing, cleaning, and part-of-speech tagging. In addition, the method still has problems such as high modeling difficulty and sparse data dimensions.

[0010] The mainstream method is the deep learning research method that combines the characteristics of the educational discipline. For example, an improved model based on BiLSTM-CRF is constructed based on the characteristics of the domain, and the root vector features of radicals are integrated for chemical entity recognition; Chinese poetry is recognized through the Bert (Bidirectional Encoder Representations from Transformers)-BiLSTM-CRF model that integrates the attention mechanism.

[0011] However, in educational texts, the vocabulary in the subject area has a certain degree of professionalism, and the above-mentioned existing entity recognition and generation methods have shortcomings in terms of the accuracy of entity recognition and generation. Summary of the invention

[0012] In order to solve the above problems existing in the prior art, the present invention provides a method and device for subject text entity extraction and keyword generation based on neural network.

[0013] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0014] In a first aspect, the present invention provides a subject text entity extraction and keyword generation method, the subject text entity extraction and keyword generation method comprising:

[0015] Generate an original text sequence according to the original text;

[0016] A global feature matrix, a local feature matrix and a head-end reinforcement feature matrix are obtained according to a plurality of character-level and word-level splicing feature vectors of the original text sequence, and a fused feature matrix is ​​obtained by performing feature fusion on the global feature matrix, the local feature matrix and the head-end reinforcement feature matrix using a multi-head self-attention mechanism, and a set of existing knowledge points is obtained by decoding the fused feature matrix; wherein the character-level and word-level splicing feature vectors are obtained by performing character-level feature extraction and word-level feature extraction on the original text sequence and then performing a splicing operation; the set of existing knowledge points includes entity knowledge points existing in the original text;

[0017] According to the original text sequence, a word embedding vector containing semantic information of the original text is generated by using BART and SoftMask, and a generated knowledge point set is obtained based on the word embedding vector; the generated knowledge point set includes entity knowledge points that do not exist in the original text and are associated with the original text;

[0018] The key knowledge point set is obtained by screening according to the word importance score, the existing knowledge point set and the generated knowledge point set.

[0019] Optionally, the multiple character-level word-level concatenation feature vectors are obtained by:

[0020] Extracting multiple word-level feature vectors from the original text sequence using BERT;

[0021] Extracting multiple word-level feature vectors from the original text sequence using a bidirectional maximum matching algorithm and one-hot encoding;

[0022] By concatenating the word-level feature vector and the character-level feature vector, multiple character-level word-level concatenated feature vectors are obtained.

[0023] Optionally, obtaining a set of existing knowledge points by decoding the fused feature matrix includes:

[0024] The fusion feature matrix is ​​decoded according to preset classification labels to obtain an existing knowledge point set consisting of entity knowledge points existing in the original text.

[0025] Optionally, a global feature matrix, a local feature matrix and a head-to-tail reinforcement feature matrix are obtained according to a plurality of character-level and word-level concatenated feature vectors of the original text sequence, including:

[0026] The global feature matrix is ​​obtained by BiGRU based on multiple character-level word-level concatenation feature vectors;

[0027] Obtaining a head-to-tail reinforcement feature matrix through two LSTM layers according to the global feature matrix and the multiple character-level word-level concatenation feature vectors;

[0028] A local feature matrix is ​​obtained according to the multiple character-level word-level concatenated feature vectors by introducing a convolution module with a self-attention mechanism.

[0029] Optionally, the BART includes a BART encoding layer and a BART decoding layer; an output end of the BART decoding layer and an output end of the BART encoding layer are both connected to a pointer generation network.

[0030] In a second aspect, the present invention provides a subject text entity extraction and keyword generation device, the subject text entity extraction and keyword generation device comprising:

[0031] An original text sequence generation module, used for generating an original text sequence according to the original text;

[0032] An existing knowledge point set determination module is used to obtain a global feature matrix, a local feature matrix and a head-end reinforcement feature matrix according to a plurality of character-level word-level splicing feature vectors of the original text sequence, perform feature fusion on the global feature matrix, the local feature matrix and the head-end reinforcement feature matrix using a multi-head self-attention mechanism to obtain a fused feature matrix, and obtain an existing knowledge point set by decoding the fused feature matrix; wherein the character-level word-level splicing feature vectors are obtained by performing character-level feature extraction and word-level feature extraction on the original text sequence and then performing a splicing operation; the existing knowledge point set includes entity knowledge points existing in the original text;

[0033] A generated knowledge point set determination module is used to generate a word embedding vector containing the semantic information of the original text according to the original text sequence using BART and SoftMask, and obtain a generated knowledge point set based on the word embedding vector; the generated knowledge point set includes entity knowledge points that do not exist in the original text and are associated with the original text;

[0034] The key knowledge point set screening module is used to screen the key knowledge point set according to the word importance score, the existing knowledge point set and the generated knowledge point set to obtain the key knowledge point set.

[0035] Optionally, the multiple character-level word-level concatenation feature vectors are obtained by:

[0036] Extracting multiple word-level feature vectors from the original text sequence using BERT;

[0037] Extracting multiple word-level feature vectors from the original text sequence using a bidirectional maximum matching algorithm and one-hot encoding;

[0038] By concatenating the word-level feature vector and the character-level feature vector, multiple character-level word-level concatenated feature vectors are obtained.

[0039] Optionally, the existing knowledge point set determination module obtains the existing knowledge point set by decoding the fusion feature matrix, including:

[0040] The fusion feature matrix is ​​decoded according to preset classification labels to obtain an existing knowledge point set consisting of entity knowledge points existing in the original text.

[0041] Optionally, the existing knowledge point set determination module obtains a global feature matrix, a local feature matrix and a head-to-tail reinforcement feature matrix according to a plurality of character-level word-level concatenated feature vectors of the original text sequence, including:

[0042] The global feature matrix is ​​obtained by BiGRU based on multiple character-level word-level concatenation feature vectors;

[0043] Obtaining a head-to-tail reinforcement feature matrix through two LSTM layers according to the global feature matrix and the multiple character-level word-level concatenation feature vectors;

[0044] A local feature matrix is ​​obtained according to the multiple character-level word-level concatenated feature vectors by introducing a convolution module with a self-attention mechanism.

[0045] Optionally, the BART includes a BART encoding layer and a BART decoding layer; an output end of the BART decoding layer and an output end of the BART encoding layer are both connected to a pointer generation network.

[0046] In a subject text entity extraction and keyword generation method provided by the present invention, there is a knowledge point set obtained by decoding a fusion feature matrix, and the fusion feature matrix is ​​obtained by obtaining a global feature matrix, a local feature matrix and a head-tail reinforcement feature matrix based on multiple character-level word-level splicing feature vectors of an original text sequence. Therefore, the present invention introduces information from multiple angles in the process of entity knowledge point recognition, thereby improving the accuracy of entity recognition. BART can be used to improve the understanding of general semantic information, and the softMask mechanism is used to distinguish the importance of different words in the original text in a "soft occlusion" manner, thereby enhancing the ability to perceive semantic information and the ability to control semantics, and improving the accuracy of generating entity knowledge points that do not exist in the original text and are associated with the original text. Under the condition that the accuracy of the existing knowledge point set and the generated knowledge point set is higher, a more accurate and comprehensive set of key knowledge points is obtained.

[0047] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flowchart of a subject text entity extraction and keyword generation method provided by an embodiment of the present invention;

[0049] Figure 2 is a schematic diagram of the structure of an entity extraction module provided by an embodiment of the present invention;

[0050] Figure 3 is a schematic diagram of the structure of a knowledge generation module provided by an embodiment of the present invention;

[0051] Figure 4 It is a schematic diagram of a process of applying the SoftMask technology to perform word embedding according to an embodiment of the present invention;

[0052] Figure 5 It is a schematic diagram of a process for obtaining a set of key knowledge points provided by an embodiment of the present invention;

[0053] Figure 6It is a structural diagram of a subject text entity extraction and keyword generation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0055] In order to solve the problem of low accuracy of existing entity recognition and generation methods in the field of education, the embodiment of the present invention provides a subject text entity extraction and keyword generation method, see Figure 1 , Figure 1 The flowchart of a subject text entity extraction and keyword generation method provided by an embodiment of the present invention specifically includes the following steps:

[0056] Step S101, generating an original text sequence according to the original text.

[0057] In the embodiment of the present invention, the original text refers to the natural language text that is directly input. For example, "a process is the basic unit for resource allocation and scheduling in the system". The original text sequence refers to the original text converted into a processable form. Generally, the original text can be decomposed into a word-level sequence.

[0058] Step S102, obtaining a global feature matrix, a local feature matrix and a head-to-tail reinforcement feature matrix based on multiple character-level and word-level splicing feature vectors of the original text sequence, performing feature fusion on the global feature matrix, the local feature matrix and the head-to-tail reinforcement feature matrix using a multi-head self-attention mechanism to obtain a fused feature matrix, and obtaining a set of existing knowledge points by decoding the fused feature matrix; wherein the character-level and word-level splicing feature vectors are obtained by performing character-level feature extraction and word-level feature extraction on the original text sequence and then using a splicing operation; the set of existing knowledge points includes entity knowledge points existing in the original text.

[0059] In an embodiment of the present invention, a specific method for obtaining a character-level word-level splicing feature vector includes: extracting features for each Chinese character or character from an original text sequence to obtain multiple character-level feature vectors, then extracting features for each word from an original text sequence to obtain multiple word-level feature vectors, and then splicing byte feature vectors and word-level feature vectors to obtain multiple character-level word-level splicing feature vectors. Since the character-level word-level splicing feature vector can reflect both character-level information and word-level information, the information it represents is richer.

[0060] In an embodiment of the present invention, a global feature matrix, a local feature matrix, and a head-end enhancement feature matrix can be obtained based on multiple character-level word-level splicing feature vectors. Among them, the global feature matrix contains the global information of the entire original text sequence, the local feature matrix contains the local context information of the original text sequence, and the head-end enhancement feature matrix is ​​obtained by enhancing the features of the beginning and end parts of each entity, which is intended to capture the boundary information of each entity. In an embodiment of the present invention, an entity refers to a word with a specific meaning in a text, and specifically can be a proper noun in an educational field text.

[0061] The use of a multi-head self-attention mechanism can be used to more comprehensively capture the potential multiple semantic associations in a text sequence. Therefore, in an embodiment of the present invention, a multi-head self-attention mechanism is used to fuse the global feature matrix, the local feature matrix, and the head-to-tail reinforcement feature matrix to obtain a fused feature matrix.

[0062] In an embodiment of the present invention, obtaining the existing knowledge point set by decoding the fused feature matrix includes decoding the fused feature matrix according to a preset classification label to obtain the existing knowledge point set consisting of entity knowledge points existing in the original text.

[0063] The preset classification labels are standards for guiding decoding, and the preset classification labels may include entity labels and non-entity labels. Among them, entity labels are used to identify the parts of the text that are real knowledge entities. Non-entity labels are used to identify other general words or functional words in the text, such as conjunctions, auxiliary words, etc. Therefore, by decoding the fusion feature matrix according to the preset classification labels, a set of existing knowledge points consisting of entity knowledge points existing in the original text is obtained.

[0064] Step S103, based on the original text sequence, using BART and SoftMask to generate a word embedding vector containing the semantic information of the original text, and based on the word embedding vector, obtain a generated knowledge point set; the generated knowledge point set includes entity knowledge points that do not exist in the original text and are associated with the original text.

[0065] BART is a denoising autoencoder built using a sequence-to-sequence model. BART can be used to embed words into the original text sequence and introduce the SoftMask method, which allows the model to make more accurate predictions based on contextual information while retaining the word vector representation of the original word. SoftMask can be used to weight word embeddings and selectively focus on certain words or parts, such as entities. By assigning a weight to each word, the accuracy of predictions can be improved based on the importance of the word or the contextual relationship.

[0066] Based on this, according to BART and SoftMask, a word embedding vector containing the semantic information of the original text can be generated according to the original text sequence, and based on the word embedding vector with high weight, entity knowledge points that do not exist in the original text and are associated with the original text can be generated according to methods such as similarity calculation to obtain a set of generated knowledge points.

[0067] Step S104, filtering to obtain a key knowledge point set according to the word importance score, the existing knowledge point set and the generated knowledge point set.

[0068] In the embodiment of the present invention, the importance degree between entity knowledge points can be obtained according to the word importance score, the existing knowledge point set and the generated knowledge point set, and the key knowledge point set can be screened out.

[0069] In an embodiment of the present invention, the existing knowledge point set is obtained by decoding a fused feature matrix, and the fused feature matrix is ​​obtained by obtaining a global feature matrix, a local feature matrix, and a head-to-tail reinforcement feature matrix based on multiple character-level word-level splicing feature vectors of the original text sequence. Therefore, the embodiment of the present invention introduces information from multiple angles in the process of entity knowledge point recognition, thereby improving the accuracy of entity recognition. BART can be used to improve the understanding of general semantic information, and the softMask mechanism is used to distinguish the importance of different words in the original text in a "soft masking" manner, thereby enhancing the ability to perceive semantic information and the ability to control semantics, and improving the accuracy of generating entity knowledge points that do not exist in the original text and are associated with the original text. Under the condition that the accuracy of the existing knowledge point set and the generated knowledge point set is higher, a more accurate and comprehensive set of key knowledge points is obtained.

[0070] In one implementation, the specific steps of obtaining the set of existing knowledge points can be implemented by an entity extraction module, see Figure 2 , Figure 2 : is a schematic diagram of the structure of the entity extraction module provided by an embodiment of the present invention. The entity extraction module may include an input layer, an educational information enhancement unit, a feature extraction layer, a multi-head self-attention layer, a decoding layer, and an output layer. The specific process of entity extraction based on the entity extraction module is as follows:

[0071] The original text sequence is input into the input layer. The original text sequence is obtained according to the original text. For example, the original text can be: Process is the basic unit for resource allocation and scheduling in the system. The original text sequence is X = {x1, x2, x3, ..., x n}, i = 1, 2, ..., n, n is the total number of words in the original text. When the original text is a process, which is the basic unit for resource allocation and scheduling in the system, n = 19.

[0072] In an embodiment of the present invention, BERT and a bidirectional maximum matching algorithm are introduced into the educational information enhancement unit.

[0073] In the embodiment of the present invention, multiple character-level word-level concatenation feature vectors are obtained by:

[0074] Use BERT to extract multiple word-level feature vectors from the original text sequence;

[0075] Multiple word-level feature vectors are extracted from the original text sequence using a bidirectional maximum matching algorithm and one-hot encoding;

[0076] By concatenating the word-level feature vector and the character-level feature vector, multiple character-level word-level concatenated feature vectors are obtained.

[0077] For the input original text sequence X = {x1, x2, x3, ..., x n}, using BERT to extract word-level features can be based on x i Generate word-level feature vector t with rich information i .

[0078] By using the bidirectional maximum matching algorithm, the input original text sequence can be segmented and matched with the preset dictionary entities. The forward matching and reverse matching results are compared, and the segmentation results with fewer segments are selected. The feature vector is constructed by one-hot encoding to obtain the dictionary feature D = d1, d2, ..., d n . i With the dictionary feature d i Concatenate to get the character-level word-level concatenation feature vector w i ,

[0079] The feature extraction layer includes a global feature extraction module, an entity boundary enhancement module and a local feature extraction module.

[0080] In the embodiment of the present invention, a global feature matrix, a local feature matrix and a head-to-tail reinforcement feature matrix are obtained according to a plurality of character-level word-level concatenation feature vectors of the original text sequence, including:

[0081] The global feature matrix is ​​obtained by BiGRU based on multiple character-level word-level concatenation feature vectors;

[0082] According to the global feature matrix and multiple character-level word-level concatenation feature vectors, the head-tail reinforcement feature matrix is ​​obtained through two LSTM layers;

[0083] According to multiple character-level word-level concatenation feature vectors, a local feature matrix is ​​obtained by introducing a convolution module with a self-attention mechanism.

[0084] The global feature extraction module includes BiGRU, i.e. bidirectional GRU (Gated Recurrent Unit). The unique structure of GRU can extract features from multiple character-level and word-level concatenated feature vectors, retaining key and useful information. The bidirectional GRU can extract input context information in both directions to enrich semantic features.

[0085] However, since the use of BiGRU will ignore some local features, a local feature extraction module is introduced in the embodiment of the present invention. The local feature extraction module includes a convolution module that introduces a self-attention mechanism. Specifically, the self-attention mechanism Self-Attention is introduced at the output end of two convolutional layers CNN of different sizes to enhance the extraction of local information to obtain a local feature matrix. The specific calculation formula of the convolutional layer is as follows:

[0086] r q =ReLU(convld(w i ));

[0087] In the formula, r q is the output result of the convolution layer, w i is the character-level word-level concatenation feature vector, convld(·) represents convolution, ReLU(·) is the nonlinear activation function, and q represents the convolution kernel size.

[0088] In the education field text, there is no explicit word boundary information. Therefore, in the embodiment of the present invention, the entity boundary information of words is enhanced to improve the model's ability to extract entity features. Specifically, two single-layer LSTM networks are introduced in the entity boundary enhancement module to predict the head entity and the tail entity, and obtain the head and tail enhanced feature matrix. Among them, LSTM start Can be used to predict head entities, LSTM end It can be used to predict the tail entity and enhance the boundary information of the entity head and tail position by integrating with global and local features. The head entity refers to the first word in an entity, and the tail entity refers to the last word in an entity.

[0089] In an embodiment of the present invention, a multi-head self-attention layer is introduced to first fuse the input global feature matrix, local feature matrix and head-to-tail reinforcement feature matrix to obtain an initial fusion matrix, and then the multi-head self-attention mechanism is used to extract features from the initial fusion matrix from three different spatial angles through three attention heads, Head1, Head2 and Head3 to obtain information feature matrices of three different spaces, and then the information feature matrices of the three different spaces are spliced ​​through the full connection layer Full Connection to obtain a fused feature matrix.

[0090] In an embodiment of the present invention, the multi-head self-attention mechanism is an improvement based on self-attention, which focuses more on obtaining the internal associations of information. Specifically, the mechanism maps the same entity multiple times. Each time it is mapped to a space, the entity is given a new meaning, that is, the data is feature extracted from different angles, and finally the information features of different spaces are spliced ​​to form the final fusion feature representation. By inputting the output of the feature extraction layer into this layer, the ability of the model to obtain key information can be indirectly improved, the waste of feature information can be avoided, and the prediction efficiency of the model can be improved.

[0091] In an embodiment of the present invention, the decoding layer can decode the fusion feature matrix according to the preset classification labels to obtain a set of existing knowledge points consisting of entity knowledge points existing in the original text. Among them, the decoding layer can be CRF (conditional random field). CRF can combine context-related constraints to proofread the correlation between labels, thereby reducing the output probability of incorrect labels. Among them, the preset classification labels can include entity labels and non-entity labels.

[0092] Finally, the output layer outputs the result, O represents the non-entity knowledge point in the original text under the non-entity label, B-Know and I-Know represent the entity knowledge point in the original text under the entity label. Specifically, B-Know represents the first word of the entity knowledge point, and I-Know represents the last word of the entity knowledge point.

[0093] The existing knowledge point set can be obtained by using B-Know and I-Know, wherein the existing knowledge point set includes entity knowledge points existing in the original text.

[0094] In an embodiment of the present invention, an educational information enhancement unit that integrates Bert word-level features and external preset dictionary entity features enhances semantic representation and is more suitable for professional field texts; and the feature extraction layer pays more attention to local features on the basis of global features and enhances the feature extraction capability by strengthening the boundary information of the first and last positions of the entity; finally, through the multi-head self-attention layer, the discriminant features of the entity are extracted from multiple angles, which helps to improve the fitting ability of the overall model, thereby improving the accuracy of subject text entity extraction.

[0095] In the embodiment of the present invention, BERT is used to fuse the context information on the left and right sides to enhance the semantic representation of the word, thereby alleviating the problem of polysemy of a word; a preset external dictionary in the field of education is built, a bidirectional maximum matching strategy is introduced, a distributed dictionary feature representation is obtained, and the recognition accuracy of the model for rare or unknown knowledge entities is improved; at the feature extraction level, global features are focused on, and a bidirectional gated recurrent unit BiGRU is introduced to better obtain the long sequence semantic features of sentence dependencies and solve the gradient vanishing problem. Compared with the use of traditional LSTM networks, GRU has fewer parameters, faster convergence speed, and smaller computational complexity than LSTM. At the same time, GRU has a relatively high utilization efficiency of the previous and next information, is not easily dominated by the close context, and is not easily ignored. Key information at a distance, so GRU is more efficient and has more advantages in global feature extraction; local features are focused on, and a self-attention mechanism is introduced on the basis of the convolutional layer to pay more attention to key knowledge-guided entity recognition; because there is no explicit word boundary information in the education field text, two single-layer LSTM networks are introduced to predict the head entity and the tail entity, obtain the head and tail features of the entity, and strengthen the entity head and tail position boundary information by integrating with global and local features. The fused results are input into the multi-head self-attention layer to extract the discriminant features of the entity from multiple angles, which helps to improve the fitting ability of the overall model. Compared with the traditional attention mechanism, it lacks the consideration of features from different angles and is easily affected by data noise; finally, the conditional random field CRF model is used to generate the global optimal sequence. Compared with the prior art, the embodiment of the present invention has better recognition of the polysemy problem in the field of education, and has a higher recognition accuracy for rare or unknown knowledge entities.

[0096] In one implementation, the specific steps of obtaining and generating a knowledge point set can be implemented by a knowledge generation module, see Figure 3 , Figure 3 : is a schematic diagram of the structure of the knowledge generation module provided by an embodiment of the present invention, and the knowledge generation module includes a word embedding layer, a BART encoding layer, a BART decoding layer, and a pointer generation network. Among them, BART is a Transformer with both contextual information and autoregressive characteristics. The specific process of obtaining and generating a knowledge point set based on the knowledge generation module is as follows:

[0097] The importance score of words can be obtained in the multi-head self-attention layer in the entity extraction module. After the sigmoid function transformation, the importance score of each word can be converted into the range of [0,1] to obtain the keyword probability score p.

[0098] In the word embedding layer, the application of SoftMask technology enables the knowledge generation module to focus on the key information in the original sentence, that is, the entity part, while reducing its attention to the non-keyword part, that is, the non-entity part. For sourcetext, the original text sequence X = {x1, x2, ..., x n}, whose length is n, and the vector representation Emb of the original text sequence X is obtained through BART word embedding technology x ={Emb x1 ,Emb x2 ,…,Emb xn Combined with the importance scores of the words obtained above, we can get the entity probability score p = {p1, p2, ..., p n}. Apply SoftMask technology and introduce a special symbol [MASK] word embedding vector Emb MASK The original meaning of the [MASK] tag is to mask the words in the actual input original text sequence, that is, to directly use the [MASK] tag to replace certain words in the input original text, thereby playing a direct masking role. Since the [MASK] tag itself is used to replace words, the word itself can also be considered as a special word. Therefore, [MASK] will exist in the vocabulary of the knowledge generation module as a "word", so [MASK] will also correspond to a word embedding representation recorded as Emb MASK , the word vector representation will also be updated as the network parameters are trained. Figure 4 , Figure 4 SoftMask is a flow chart of word embedding using the SoftMask technology provided by an embodiment of the present invention. SoftMask uses a "soft masking" form, that is, the original original sentence word vector representation Emb x , adjusted to its word vector Emb with [MASK] MASK The weights are p and (1-p), which represent the probability that each word in the original text is key knowledge, that is, an entity, and non-key knowledge, that is, a non-entity. That is, the adjusted original sentence embedding is expressed as SoftEmb, that is, the softmasked embedding of the original sentence. SoftEmb is calculated by multiplying the original sentence embedding and the corresponding elements in p, and multiplying the [MASK] embedding and the corresponding elements in (1-p), and then adding the corresponding elements. The specific calculation formula is expressed as:

[0099] SoftEmb x =p*Emb x +(1-p)*Emb MASK ;

[0100] This method of weighted masking of non-important parts based on importance scores and using [MASK] word vectors reduces the information of non-important parts. At the same time, the relative amount of information of words with a small degree of "soft masking" becomes larger. In this way, softMask distinguishes the primary and secondary parts of the original sentence in a smoother way.

[0101] In an embodiment of the present invention, the BART encoding layer is composed of 12 superimposed Transformer Encoder Layers (a neural network model structure encoder layer). This partial structure can be regarded as similar to the BERT model. Its input is text Input Embedding and Positional Encoding is added to generate word vectors for the text to ensure that each Transformer Encoder Layer can take into account the relative position information of the text to a certain extent while performing parallel operations.

[0102] Among them, the Transformer Encoder Layer consists of a multi-head attention mechanism and a fully connected layer. The multi-head attention mechanism consists of multiple self-attention mechanisms. The self-attention mechanism is based on W Q , W K and W V The three preset matrices transform the word vectors of the original text into the Q, K, and V vectors required by the self-attention mechanism, and then calculate the Attention value of the original text. The calculation formula is as follows:

[0103]

[0104] Among them, d k is the dimension of the K vector, and T represents the transpose of the matrix. In order to prevent the performance of the model from being affected by too many network layers, the model does not directly send the attention values ​​of each character to the fully connected layer after splicing, but performs multiple Add&Norm (residual connection) and Layer Norm (normalization) operations to obtain the hidden layer vector, that is, the hidden state of the BART encoding layer, and finally realizes the conversion of the input text into an embedded vector containing its semantic information.

[0105] The input of the BART decoding layer is the entity knowledge points existing in the original text, namely generating words, and the output is the hidden state of the BART decoding layer. The output of the BART decoding layer and the output of the BART encoding layer are both connected to the pointer generation network. After the BART decoding layer outputs the BART decoding layer hidden state, in the pointer generation network, the BART decoding layer hidden state is combined with the BART encoding layer hidden state to generate an attention distribution, namely the context vector. The decoding layer hidden state is combined with the context vector and fed through two linear layers to generate a vocabulary distribution. After that, the decoding layer hidden state will be similarity calculated with the BART encoding layer hidden state of the original text vocabulary pointed to by its corresponding attention distribution, and then the generation probability parameter P of the PGN module is calculated through this similarity and other parameters. gen , and finally decides whether to generate the vocabulary. Since the distribution of the word vectors of the BART model in space is not uniform, but conical, the cosine similarity calculation method cannot be simply used. The word vector coordinate basis generated by BART is not a standard orthogonal basis, that is, there is anisotropy, and the word vector needs to be converted into a standard orthogonal basis. The embodiment of the present invention selects the whitening (BERT-whitening) method, which is to convert all word vectors into vectors with a mean of 0 and a covariance matrix as a unit matrix.

[0106] When the similarity is high enough, the value of the generation probability parameter will increase accordingly, and the knowledge generation module will be more inclined to retain the generated vocabulary. Otherwise, the model will be more inclined to copy the vocabulary in the original text. In this way, when the knowledge generation module generates vocabulary with different semantics from the original text, it will discard the vocabulary and copy the vocabulary of the original text instead. Finally, the attention distribution and vocabulary distribution generate the final distribution under the action of the copy mechanism, and obtain the generated knowledge point set, which improves the accuracy and stability of the knowledge generation module in generating text summaries. Among them, 1-P gen represents the non-keyword probability score, that is, the non-entity probability score, P gen Represents the keyword probability score, that is, the entity probability score.

[0107] In the embodiment of the present invention, the importance score of each word in the original text is obtained as the auxiliary keyword information of the knowledge generation module; then the keyword information is combined with the softMask mechanism to distinguish the primary and secondary parts of the original text sentence in a "soft masking" manner, thereby realizing the semantic information perception ability of the knowledge generation model and enhancing the semantic control ability; the large language pre-training BART generation model is used to improve the understanding of general semantic information, and at the same time, the pointer generation network is integrated after the BART decoding layer to reduce the probability of the knowledge generation model generating semantically incorrect vocabulary that is inconsistent with the original text. Compared with the prior art, the embodiment of the present invention can pay more attention to the importance of text words for knowledge generation in the field of education, and at the same time can reduce the probability of generating semantically incorrect vocabulary that is inconsistent with the original text.

[0108] See also Figure 5 , Figure 5 It is a flow chart of obtaining a key knowledge point set provided by an embodiment of the present invention. For the original text data set, i.e., the original text sequence, after obtaining an existing knowledge point set through knowledge recognition and obtaining a generated knowledge point set through knowledge generation, the existing knowledge point set is combined with the word importance score as the first part of the input, and the generated knowledge point set is combined with the attention score as the second part of the input for knowledge fusion. Then, based on the existing knowledge point set and the word importance score, the importance of words under different classification labels relative to the subject text words can be distinguished; based on the generated knowledge point set and the attention score, the importance of the generated knowledge points can be distinguished, so that a key knowledge information set, i.e., a key knowledge point set, is obtained through screening. Among them, the word importance score is extracted by fusing the global feature matrix, the local feature matrix, and the head-to-tail reinforcement feature matrix from the multi-head self-attention layer to obtain a fused feature matrix.

[0109] By assigning different importance coefficients to different classification labels, for example, theorem knowledge and generated knowledge have higher importance coefficients, while experimental equipment knowledge is slightly less important. Then the existing knowledge point set and the generated knowledge point set are screened and filtered, and the knowledge points with high similarity in the two sets are selected by combining the minimum edit distance similarity with the Jaccard similarity judgment method. The selected knowledge points appear in both sets at the same time, indicating that they are more important than other knowledge in the subject text, so these knowledge points are assigned a higher importance coefficient. For other knowledge point information that has not been selected, the relative importance is calculated based on the importance coefficients of different categories combined with their own importance scores or attention scores. Then, after the last step of screening, the knowledge points with extremely low importance are removed, and the final set of key knowledge points is formed through knowledge fusion.

[0110] Based on the same inventive concept, the embodiment of the present invention also provides a subject text entity extraction and keyword generation device, see Figure 6 , Figure 6 1 is a schematic diagram of a structure of a subject text entity extraction and keyword generation device provided by an embodiment of the present invention, the subject text entity extraction and keyword generation device comprises:

[0111] The original text sequence generating module 601 is used to generate an original text sequence according to the original text;

[0112] The existing knowledge point set determination module 602 is used to obtain a global feature matrix, a local feature matrix and a head-end reinforcement feature matrix according to multiple character-level word-level splicing feature vectors of the original text sequence, and to fuse the global feature matrix, the local feature matrix and the head-end reinforcement feature matrix using a multi-head self-attention mechanism to obtain a fused feature matrix, and to obtain an existing knowledge point set by decoding the fused feature matrix; wherein the character-level word-level splicing feature vector is obtained by performing character-level feature extraction and word-level feature extraction on the original text sequence and then using a splicing operation; the existing knowledge point set includes entity knowledge points existing in the original text;

[0113] A generated knowledge point set determination module 603 is used to generate a word embedding vector containing semantic information of the original text using BART and SoftMask according to the original text sequence, and obtain a generated knowledge point set based on the word embedding vector; the generated knowledge point set includes entity knowledge points that do not exist in the original text and are associated with the original text;

[0114] The key knowledge point set screening module 604 is used to screen the key knowledge point set according to the word importance score, the existing knowledge point set and the generated knowledge point set.

[0115] In an embodiment of the present invention, the existing knowledge point set is obtained by decoding the fused feature matrix, and the fused feature matrix is ​​obtained by obtaining a global feature matrix, a local feature matrix, and a head-tail reinforcement feature matrix based on multiple character-level word-level splicing feature vectors of the original text sequence. Therefore, the embodiment of the present invention introduces information from multiple angles in the process of entity knowledge point recognition, thereby improving the accuracy of entity recognition. BART can be used to improve the understanding of general semantic information, and the softMask mechanism is used to distinguish the importance of different words in the original text in a "soft occlusion" manner, thereby enhancing the ability to perceive semantic information and the ability to control semantics, and improving the accuracy of generating entity knowledge points that do not exist in the original text but are associated with the original text. Under the condition that the accuracy of the existing knowledge point set and the generated knowledge point set is higher, a more accurate and comprehensive set of key knowledge points is obtained.

[0116] Optionally, the multiple character-level word-level concatenation feature vectors are obtained by:

[0117] Extracting multiple word-level feature vectors from the original text sequence using BERT;

[0118] Extracting multiple word-level feature vectors from the original text sequence using a bidirectional maximum matching algorithm and one-hot encoding;

[0119] By concatenating the word-level feature vector and the character-level feature vector, multiple character-level word-level concatenated feature vectors are obtained.

[0120] Optionally, the existing knowledge point set determination module obtains the existing knowledge point set by decoding the fusion feature matrix, including:

[0121] The fusion feature matrix is ​​decoded according to preset classification labels to obtain an existing knowledge point set consisting of entity knowledge points existing in the original text.

[0122] Optionally, the existing knowledge point set determination module obtains a global feature matrix, a local feature matrix and a head-to-tail reinforcement feature matrix according to a plurality of character-level word-level concatenated feature vectors of the original text sequence, including:

[0123] The global feature matrix is ​​obtained by BiGRU based on multiple character-level word-level concatenation feature vectors;

[0124] Obtaining a head-to-tail reinforcement feature matrix through two LSTM layers according to the global feature matrix and the multiple character-level word-level concatenation feature vectors;

[0125] A local feature matrix is ​​obtained according to the multiple character-level word-level concatenated feature vectors by introducing a convolution module with a self-attention mechanism.

[0126] Optionally, the BART includes a BART encoding layer and a BART decoding layer; an output end of the BART decoding layer and an output end of the BART encoding layer are both connected to a pointer generation network.

[0127] It should be noted that the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention.

[0128] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0129] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other changes to the disclosed embodiments by viewing the drawings and the disclosed content. In the description of the present invention, the term "comprising" does not exclude other components or steps, "one" or "an" does not exclude multiple situations, and "multiple" means two or more, unless otherwise clearly and specifically limited. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0130] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0131] It should be noted that the device of the embodiment of the present invention is a device that applies the above-mentioned neural network-based subject text entity extraction and keyword generation method. All embodiments of the above-mentioned neural network-based subject text entity extraction and keyword generation method are applicable to the device and can achieve the same or similar beneficial effects.

[0132] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A subject text entity extraction and keyword generation method, characterized in that: The subject text entity extraction and keyword generation method includes: Generate an original text sequence according to the original text; A global feature matrix, a local feature matrix and a head-end reinforcement feature matrix are obtained according to a plurality of character-level and word-level splicing feature vectors of the original text sequence, and a fused feature matrix is ​​obtained by performing feature fusion on the global feature matrix, the local feature matrix and the head-end reinforcement feature matrix using a multi-head self-attention mechanism, and a set of existing knowledge points is obtained by decoding the fused feature matrix; wherein the character-level and word-level splicing feature vectors are obtained by performing character-level feature extraction and word-level feature extraction on the original text sequence and then performing a splicing operation; the set of existing knowledge points includes entity knowledge points existing in the original text; According to the original text sequence, a word embedding vector containing semantic information of the original text is generated by using BART and SoftMask, and a generated knowledge point set is obtained based on the word embedding vector; the generated knowledge point set includes entity knowledge points that do not exist in the original text and are associated with the original text; The key knowledge point set is obtained by screening according to the word importance score, the existing knowledge point set and the generated knowledge point set.

2. The subject text entity extraction and keyword generation method according to claim 1 is characterized in that: The multiple character-level word-level concatenation feature vectors are obtained in the following manner: Extracting multiple word-level feature vectors from the original text sequence using BERT; Extracting multiple word-level feature vectors from the original text sequence using a bidirectional maximum matching algorithm and one-hot encoding; By concatenating the word-level feature vector and the character-level feature vector, multiple character-level word-level concatenated feature vectors are obtained.

3. The subject text entity extraction and keyword generation method according to claim 1 is characterized in that: The existing knowledge point set is obtained by decoding the fusion feature matrix, including: The fusion feature matrix is ​​decoded according to preset classification labels to obtain an existing knowledge point set consisting of entity knowledge points existing in the original text.

4. The subject text entity extraction and keyword generation method according to claim 1 is characterized in that: A global feature matrix, a local feature matrix and a head-to-tail reinforcement feature matrix are obtained according to a plurality of character-level word-level concatenated feature vectors of the original text sequence, including: The global feature matrix is ​​obtained by BiGRU based on multiple character-level word-level concatenation feature vectors; Obtaining a head-to-tail reinforcement feature matrix through two LSTM layers according to the global feature matrix and the multiple character-level word-level concatenation feature vectors; A local feature matrix is ​​obtained according to the multiple character-level word-level concatenated feature vectors by introducing a convolution module with a self-attention mechanism.

5. The subject text entity extraction and keyword generation method according to claim 1 is characterized in that: The BART includes a BART encoding layer and a BART decoding layer; an output end of the BART decoding layer and an output end of the BART encoding layer are both connected to a pointer generation network.

6. A subject text entity extraction and keyword generation device, characterized in that: The subject text entity extraction and keyword generation device comprises: An original text sequence generation module, used for generating an original text sequence according to the original text; An existing knowledge point set determination module is used to obtain a global feature matrix, a local feature matrix and a head-end reinforcement feature matrix according to a plurality of character-level word-level splicing feature vectors of the original text sequence, perform feature fusion on the global feature matrix, the local feature matrix and the head-end reinforcement feature matrix using a multi-head self-attention mechanism to obtain a fused feature matrix, and obtain an existing knowledge point set by decoding the fused feature matrix; wherein the character-level word-level splicing feature vectors are obtained by performing character-level feature extraction and word-level feature extraction on the original text sequence and then performing a splicing operation; the existing knowledge point set includes entity knowledge points existing in the original text; A generated knowledge point set determination module is used to generate a word embedding vector containing the semantic information of the original text according to the original text sequence using BART and SoftMask, and obtain a generated knowledge point set based on the word embedding vector; the generated knowledge point set includes entity knowledge points that do not exist in the original text and are associated with the original text; The key knowledge point set screening module is used to screen the key knowledge point set according to the word importance score, the existing knowledge point set and the generated knowledge point set to obtain the key knowledge point set.

7. The subject text entity extraction and keyword generation device according to claim 6 is characterized in that: The multiple character-level word-level concatenation feature vectors are obtained in the following manner: Extracting multiple word-level feature vectors from the original text sequence using BERT; Extracting multiple word-level feature vectors from the original text sequence using a bidirectional maximum matching algorithm and one-hot encoding; By concatenating the word-level feature vector and the character-level feature vector, multiple character-level word-level concatenated feature vectors are obtained.

8. The subject text entity extraction and keyword generation device according to claim 6 is characterized in that: The existing knowledge point set determination module obtains the existing knowledge point set by decoding the fusion feature matrix, including: The fusion feature matrix is ​​decoded according to preset classification labels to obtain an existing knowledge point set consisting of entity knowledge points existing in the original text.

9. The subject text entity extraction and keyword generation device according to claim 6 is characterized in that: The existing knowledge point set determination module obtains a global feature matrix, a local feature matrix and a head-to-tail reinforcement feature matrix according to a plurality of character-level word-level concatenation feature vectors of the original text sequence, including: The global feature matrix is ​​obtained by BiGRU based on multiple character-level word-level concatenation feature vectors; Obtaining a head-to-tail reinforcement feature matrix through two LSTM layers according to the global feature matrix and the multiple character-level word-level concatenation feature vectors; A local feature matrix is ​​obtained according to the multiple character-level word-level concatenated feature vectors by introducing a convolution module with a self-attention mechanism.

10. The subject text entity extraction and keyword generation device according to claim 6, characterized in that: The BART includes a BART encoding layer and a BART decoding layer; an output end of the BART decoding layer and an output end of the BART encoding layer are both connected to a pointer generation network.

Images